Speed2, a comparative speed test through different development environments

[RobertoBianchi]

Hello,

as you know we are always looking how to optimize ThinBASIC's performance.
So we changed a bit our 'classic' Test_Speed2.tbasic script and ported it on different compilers and intepreters.
Today tests are build whit Visual C#, Power Basic, Visual Basic Script and of course ThinBASIC (you'll find all sources and executables in the attached files).
Here are a result for one million FOR ... NEXT cycle :
Visual C#0.00377Power Basic0.03200ThinBASIC1.42200Visual Basic Script1.96880

Let me say amazing, .NET compiler generate light speed code, I think that will be very hard trying to reach it whit the POWER BASIC rocket.

Of course this is only a simple (and I hope not wrong) test but ... very impressive and we just want to share with all of you.
Please feel free to let us know your comments, suggestions and corrections.

Bye,
Roberto

[Petr Schreiber]

Hi Roberto,

fantastic to see VBScript is slayed by thinBASIC ( on my PC by similar ratio as in your case ).

But the PB test is unfair! TIMER has terrible precision.

Following uses QueryPerformanceCounter + REGISTER variable:


#COMPILE EXE
#DIM ALL

#INCLUDE "win32api.inc"

FUNCTION PBMAIN () AS LONG
REGISTER counter AS LONG
DIM n_str AS DOUBLE
DIM n_lvl AS DOUBLE
DIM n_wp AS DOUBLE
DIM n_vit AS DOUBLE
DIM n_adef AS DOUBLE
DIM result AS DOUBLE
DIM div AS DOUBLE

DIM MaxCount AS LONG
DIM lResult AS LONG
DIM T1 AS QUAD
DIM T2 AS QUAD
DIM I AS LONG
DIM Message AS STRING

DIM QPF AS QUAD
QueryPerformanceFrequency(QPF)

n_str = 10000
n_lvl = 10000
n_wp = 10000
n_vit = 10000
n_adef = 10000
MaxCount = 1000000

Message = "This program will solve a simple math expression " & STR$(MaxCount) & " times." + $CRLF + "Please press Yes to go on, NO to Stop"
lResult = MSGBOX(Message, %MB_YESNO, "Continue?")
IF lResult <> %IDYES THEN
EXIT FUNCTION
END IF

i = 0
div = 2

QueryPerformanceCounter(T1)
result = 0
FOR counter = 1 TO MaxCount
INCR i
result=(((n_str*85+n_lvl)*n_wp)-((n_vit*90+n_adef)*n_lvl))/div
NEXT
QueryPerformanceCounter(T2)

MSGBOX(FORMAT$((T2 - T1)/QPF, "#0.000000") & $CRLF & STR$(result) & $CRLF & "i:" & STR$(i))

END FUNCTION


This way I get 0.007 for PB and 0.004 for C#, which is not so bad ( although it is still a bit different from what I would like to see :'( ).

Regading the test method - evaluated expression does not include Counter or i variables, so it can be that C# optimizes this case as "unnecessary looping" and expression is evaluated just once. I would be for something like:


result = 0
FOR counter = 1 TO MaxCount
INCR i
result=(((n_str*85+n_lvl)*n_wp*i) -((n_vit*90+n_adef)*n_lvl*counter))/div
NEXT


After such a change thinBASIC gets even more speed advance from VBS, at least on my PC:
VBS: 3.281
thinBASIC: 2.594
PB: 0.011
C#: 0.007

So ... C# probably does not cheat so much as I thought :) I wonder how they make it that fast? Friend does work in C# and the proggies does not seems to be deadly fast ???

But what I really like is tB over VBS :)


Bye,
Petr

[ErosOlmi]

Well, I'm happy problem about Power Basic was from our side.
I didn't think about possible compiler pre-compile code optimizations and the fact that code inside FOR/NEXt is constant.

Remain the fact that C# seem very fast.

Thanks Petr.
Eros

[Michael Clease]

Would be interesting to see a freebasic version if anyone knows the syntax.

[José Roca]

I wonder how they make it that fast?


Probably they will reduce the precision control bits of the FPU for the accuracy of the mantissa from 64 bits to 52 bits.

[RobertoBianchi]

Hi,

just to adhere on changes rightly made by Petr to POWER BASIC code, I have slightly changed the C# code that carryover below:


using System;
using System.Runtime.InteropServices;
using System.ComponentModel;
using System.Threading;

namespace ConsoleApplication1
{
class Program
{
[DllImport("User32.dll")]
private static extern int MessageBox(int h, string m, string c, int type);
[DllImport("Kernel32.dll")]
private static extern bool QueryPerformanceCounter(out long lpPerformanceCount);
[DllImport("Kernel32.dll")]
private static extern bool QueryPerformanceFrequency(out long lpFrequency);

public const Int32 MB_OK = 0;
public const Int32 MB_YESNO = 4;
public const Int32 IDYES = 6;

static void Main(string&#91;] args)
{
double n_str = 10000;
double n_lvl = 10000;
double n_wp = 10000;
double n_vit = 10000;
double n_adef = 10000;
double result = 10000;
double div = 2;
Int32 counter = 0;
Int32 MaxCount = 1000000;
Int32 lResult = 0;
Int32 i = 0;
long last;
long now;
long freq;

string Message = "This program will solve a simple math expression " + MaxCount + " times.\n" + "Please press Yes to go on, NO to Stop";

lResult = MessageBox(0, Message, "Continue?", MB_YESNO);
if(lResult == IDYES)
{
if (QueryPerformanceFrequency(out freq) == false)
{
throw new Win32Exception(); // timer not supported
}
else
{
QueryPerformanceCounter(out last);
result = 0;
for (counter = 1; counter <= MaxCount; counter++)
{
++i;
result = (((n_str * 85 + n_lvl) * n_wp) - ((n_vit * 90 + n_adef) * n_lvl)) / div;
}
QueryPerformanceCounter(out now);
MessageBox(0, "Time Elapsed: " + ((double)(now - last) / (double)freq).ToString() + "\nResult: " + result.ToString() + "\ni: " + i.ToString(), "Speed Test Result", MB_OK);
}
}

}
}
}

After these changes the situation on my PC is as follows:

Visual C#0.00379Power Basic0.00828

José, interesting observation. I'll try to detect how C# works. Thanks.

Roberto

[RobertoBianchi]

Additional info:

switching on or off the C# compiler optimisation seems not changing the program performance.

[RobertoBianchi]

Again changed both PB and C# code in order to be sure that the calculation's result ins't a constant.

POWER BASIC:

#COMPILE EXE
#DIM ALL

#INCLUDE "win32api.inc"

FUNCTION PBMAIN () AS LONG
REGISTER counter AS LONG
DIM n_str AS DOUBLE
DIM n_lvl AS DOUBLE
DIM n_wp AS DOUBLE
DIM n_vit AS DOUBLE
DIM n_adef AS DOUBLE
DIM result AS DOUBLE
DIM div AS DOUBLE

DIM MaxCount AS LONG
DIM lResult AS LONG
DIM T1 AS QUAD
DIM T2 AS QUAD
DIM I AS LONG
DIM Message AS STRING

DIM QPF AS QUAD
QueryPerformanceFrequency(QPF)

n_str = 10000
n_lvl = 10000
n_wp = 10000
n_vit = 10000
n_adef = 10000
MaxCount = 1000000

Message = "This program will solve a simple math expression " & STR$(MaxCount) & " times." + $CRLF + "Please press Yes to go on, NO to Stop"
lResult = MSGBOX(Message, %MB_YESNO, "Continue?")
IF lResult <> %IDYES THEN
EXIT FUNCTION
END IF

i = 0
div = 2
result = 0

QueryPerformanceCounter(T1)
FOR counter = 1 TO MaxCount
INCR i
result=(((n_str*85+n_lvl)*n_wp)-((n_vit*90+n_adef)*n_lvl))/div * i / 123.4567
NEXT
QueryPerformanceCounter(T2)

MSGBOX(FORMAT$((T2 - T1)/QPF, "#0.000000") & $CRLF & STR$(result) & $CRLF & "i:" & STR$(i))

END FUNCTION


C#:

using System;
using System.Runtime.InteropServices;
using System.ComponentModel;
using System.Threading;

namespace ConsoleApplication1
{
class Program
{
[DllImport("User32.dll")]
private static extern int MessageBox(int h, string m, string c, int type);
[DllImport("Kernel32.dll")]
private static extern bool QueryPerformanceCounter(out long lpPerformanceCount);
[DllImport("Kernel32.dll")]
private static extern bool QueryPerformanceFrequency(out long lpFrequency);

public const Int32 MB_OK = 0;
public const Int32 MB_YESNO = 4;
public const Int32 IDYES = 6;

static void Main(string&#91;] args)
{
double n_str = 10000;
double n_lvl = 10000;
double n_wp = 10000;
double n_vit = 10000;
double n_adef = 10000;
double result = 10000;
double div = 2;
Int32 counter = 0;
Int32 MaxCount = 1000000;
Int32 lResult = 0;
Int32 i = 0;
long last;
long now;
long freq;

string Message = "This program will solve a simple math expression " + MaxCount + " times.\n" + "Please press Yes to go on, NO to Stop";

lResult = MessageBox(0, Message, "Continue?", MB_YESNO);
if(lResult == IDYES)
{
if (QueryPerformanceFrequency(out freq) == false)
{
throw new Win32Exception(); // timer not supported
}
else
{
result = 0;
QueryPerformanceCounter(out last);
for (counter = 1; counter <= MaxCount; counter++)
{
++i;
result = (((n_str * 85 + n_lvl) * n_wp) - ((n_vit * 90 + n_adef) * n_lvl)) / div * i / 123.4567;
}
QueryPerformanceCounter(out now);
MessageBox(0, "Time Elapsed: " + ((double)(now - last) / (double)freq).ToString() + "\nResult: " + result.ToString() + "\ni: " + i.ToString(), "C# Speed Test Result", MB_OK);
}
}

}
}
}

Please see results on picture attached below.

And if also wanted to consider the executable size, here are my build:
C# SpeedTest2.exe 5632 bytes
PB SpeedTest2-Petr.exe 15872 bytes

Roberto

[ErosOlmi]

Well,

now without TIMER overloading and inserting the looper variable inside the loop it seems a more real life test.
Again C# give great results. Impressed and something to consider.

Thanks Roberto.
Eros

[Petr Schreiber]

Interesting :)

Although comparing app sizes is a bit tricky as C# app means perfect 5kB but + 20MB of .NET.
I am curious if José will come with some PB/C# mantisa differences.

It is possible to kick down the PB filesize to 8,192 bytes ( FORMAT$ out, %USEMACROS = 1 in :) ):


#COMPILE EXE
#DIM ALL

%USEMACROS = 1
#INCLUDE "win32api.inc"

FUNCTION PBMAIN () AS LONG
REGISTER counter AS LONG
LOCAL n_str, n_lvl, n_wp, n_vit, n_adef, result, div AS DOUBLE
LOCAL MaxCount, lResult, i AS LONG
LOCAL T1, T2 AS QUAD
LOCAL Message AS STRING
LOCAL QPF AS QUAD

QueryPerformanceFrequency(QPF)

n_str = 10000
n_lvl = 10000
n_wp = 10000
n_vit = 10000
n_adef = 10000
MaxCount= 1000000

Message = "This program will solve a simple math expression " & STR$(MaxCount) & " times." + $CRLF + "Please press Yes to go on, NO to Stop"
lResult = MSGBOX(Message, %MB_YESNO, "Continue?")
IF lResult = %IDNO THEN EXIT FUNCTION

i = 0&
div = 2#

QueryPerformanceCounter(T1)
result = 0#
FOR counter = 1 TO MaxCount
INCR i
result=(((n_str*85+n_lvl)*n_wp)-((n_vit*90+n_adef)*n_lvl))/div
NEXT
QueryPerformanceCounter(T2)

MSGBOX(STR$((T2 - T1)/QPF) & $CRLF & STR$(result) & $CRLF & "i:" & STR$(i))

END FUNCTION


Bye,
Petr

[RobertoBianchi]

Yes I understand, but instead on my PC (with Windows XP) PB app means 8 KB + 3,31 GB of Windows. ;D

Roberto

[ErosOlmi]

Yes I understand, but instead on my PC (with Windows XP) PB app means 8 KB + 3,31 GB of Windows. ;D


To translate Roberto sentence ;D

"... also Power Basic applicactions (like almost any other programming language) needs the runtime of the operating system. Maybe a single program does not need them all but without some of them many features are not available at all. So, at the extreme, you need the full OS. More: when .Net will not be optional but integrated and hidden part of the OS, there will no room for other discussions on the need to install something ..."

[RobertoBianchi]

Petr,

I removed the error exception:
throw new Win32Exception(); // timer not supported and now the executable size is 4608 Bytes, instead UPX couldn't be used in order to reduce again the exe size.

Ciao,
Roberto

[Petr Schreiber]

Hi Roberto,

you are right. But as long as .NET 3.5 is not part of my OS... Ok I will let it be :P
I must admit I am still in a little grumpy PB did not show the back lights to C# but the opposite.

But I finally found something interesting :D - memory consumption of the testing proggies:

C# 6008 kB
VBS 5572 kB
tB 4972 kB
PB 2864 kB


Now thinBASIC slays even C# :P Thing you guys can be pretty proud of, thinking that tB parses complete text, without making it smaller using bytecode or whatever...


Petr

[ErosOlmi]

;D
Petr, you are "hard to die" man! I like it!
;D ;D

Anyhow, our tests are just very very basic and we cannot create a complete theory on them.
To be able to get some conclusions we would need to setup more advanced tests covering many different specific programming areas.
Maybe an interesting game for spare time.

Ciao
Eros

[Michael Clease]

I made a freebasic version which i think needs optimising



#include once "vbcompat.bi"
#include once "windows.bi"

#define CRLF (chr$(13) + chr$(10))
#define CTAB (chr$(9))

SetPriorityClass( GetCurrentProcess(), HIGH_PRIORITY_CLASS )

DIM n_str AS DOUBLE = 10000
DIM n_lvl AS DOUBLE = 10000
DIM n_wp AS DOUBLE = 10000
DIM n_vit AS DOUBLE = 10000
DIM n_adef AS DOUBLE = 10000
DIM result AS DOUBLE = 0
DIM counter AS LONG = 0
DIM MaxCount AS LONG = 1000000
DIm div as DOUBLE = 2
DIM i as LONG = 0
DIM Temp as iNTEGER
DIM AS ULONGint T1,T2,QPF
DIM Message AS STRING
DIM AS DOUBLE T3,t4

Temp = QueryPerformanceFrequency( cast(PLARGE_INTEGER, @QPF) )

result = 0

Temp = QueryPerformanceCounter( cast(PLARGE_INTEGER, @T1) )
FOR counter = 1 TO MaxCount
i += 1
result = (((n_str * 85 + n_lvl) * n_wp) - ((n_vit * 90 + n_adef) * n_lvl)) / div * (i - 1.0123456789012345678901234567890123456789012345678901234567890123456789) / 123.4567
NEXT
Temp = QueryPerformanceCounter( cast(PLARGE_INTEGER, @T2) )
t3 = QPF
T4 = (t2 - t1)/ T3

MessageBox( null, _
"Timer T1 : " + CTAB + str$(t1) + CRLF + _
"Timer T2 : " + CTAB + str$(t2) + CRLF + _
"Frequency: " + CTAB + str$(t3) + CRLF + _
"Seconds : " + CTAB + Format(t4,"") + CRLF + _
"Result : " + CTAB + str$(Result) + CRLF + _
"i : " + CTAB + str$(i), "FreeBASIC Speed Test Result", 0 )

SetPriorityClass( GetCurrentProcess(), NORMAL_PRIORITY_CLASS )


UPDATED SO THE TIMING WORKS

[RobertoBianchi]

Thanks Abraxas, I've been managed your FreeBASIC version of Test_Speed2 tha you find with others in the attached file.
This time I changed the calculation formula in order to force all compilers/interpretes to handle big floating point numbers as following:

result = (((n_str * 85 + n_lvl) * n_wp) - ((n_vit * 90 + n_adef) * n_lvl)) / div * (i - 1.0123456789012345678901234567890123456789012345678901234567890123456789) / 123.4567
Apparently only FreeBASIC reported an error warning for the presence of a number too large, the error was:

Test_Speed2.bas(31) warning 8(1): Literal number too big, truncated
All results are attached as picture, please note the accuracy reported by ThinBASIC.

Ciao,
Roberto

[Michael Clease]

I fixed the timing.

NO I DIDN'T

REMOVED ATTACHMENT. SEE ABOVE POST FOR UPDATED VERSION

[Petr Schreiber]

Hi,

thanks for "more cruel" and also the freebasic version.
Mike, how should I interpret 3x seconds ? Are first two times of "before" and "after" ?


Thanks,
Petr

[Michael Clease]

Sorry Petr I put the 2 extra times in for testing but didnt remove them they are the start time (t1) and Finishing time (t2).

When I get home I will look at the string handling to format the output so its not 1.71274-05 but 0.00001.71274 you get the idea.

Update

I fixed my code.

[Petr Schreiber]

Hi,

thanks for the fix!
Impressive, as fast as PB if not few % faster, 2556 kB memory occupied is also very nice.


Petr

[kryton9]

Are we looking into a freebasic or c# future?

[Michael Hartlef]

Kent, if you mean using FreeBasic as a compiler for thinBasic, then I would prefer that Eros stays with PowerBasic.

I think I should convert the sample to FreePascal or Delphi. ;D

[ErosOlmi]

Are we looking into a freebasic or c# future?


;D No.
We are just testing, experimenting, trying, making mistakes, re-doing things, change our thoughts, destroying, re-inventing, ...
In other words: we are just kids ;)

[Michael Clease]

I would prefer that Eros stays with PowerBasic.

I think I should convert the sample to FreePascal or Delphi. ;D


Whys that mike? (just curious)

and I would be interested to see the speed comparisons.

[Michael Hartlef]

Hi Mike,

I think FreeBasic is unstable. Look at the version number, even the developers don't consider it production software when you look at the number. I read so many complains about incompatibility from one update to the other. I read that they want to change the complete backend. So I think it is no tool that should be choosen to create something that has productional quality like thinBasic.

And just imagine something is not working right. Is it Eros code in thinBasic or something FreeBasic doesn't handle right. Debugging nightmare I say. If they switch the should choose a production proved environment for it.

I brought FreePascal and Delphi into the game just for fun. ;D I know that Delphi is damn fast, but I have no intentions to convince Eros to switch development tools anyway.

And when I think at the level of completeness of thinBAsic, I think that it is impossible to do the switch anyway. Can you imagine converting 125000 lines of source code? And then expect it to work without much problems? This is years of work we are talking about. And basically one or two man to do the job.

Michael

[Michael Clease]

Thanks for the info Mike.

Freebasic does seem to be missing something but I just cant put my finger on what.

I wouldn't see the point of switch development languages either but thats no ones choice apart from Eros.

thanks

Mike

[ErosOlmi]

No worry guy. We are just experimenting a bit as we usual do and I'm sure you all do too.
For us Power Basic is still the best around in terms of speed and especially in stability.

We have some ideas about to get some more very very significative speed improvement but it will require a lot of code rewriting and logic changes.
We are currently committed to create a final thinBasic version still compatible with Win9x system. It must be very rock stable because we have no intention to go back and maintain it. Done that we will move on with a complete new version no more Win9x compatible but at the same time we will change a lot in thinBasic Core engine.

Ciao
Eros

[kryton9]

Thanks for the update Eros on plans. It is really nice how you guys keep us informed and let us in on the fun behind the scenes!