Test Driven Development & Unit Testing for thinBASIC

[Petr Schreiber]

Thanks to my colleague Honza, I discovered the interesting world of Test Driven Development.

It is a topic for fat book, but let's sum it up quickly:

test driven development is approach to write safe code with predicable output
it is based on idea of writing failing tests first and implementing functionality later, to remove the errors step by step


While I hope we will be able to put together built in Unit Testing system with Eros (for ThinBASIC 2.x), let's experiment now!

I prepaired simple unitTesting.tBasicU, which covers most of the needs for TDD beginner (bold statement, ain't it).

It features engine for setting up the tests:

ut_Initialize
ut_LaunchTests
ut_SaveLog
ut_Release
ut_GetLastError


Many pre-defined asserts for typical cases:

ut_assertEqual
ut_assertEqualRounded

ut_assertEqualEpsilon
ut_assertNotEqual
ut_assertEqualText
ut_assertNotEqualText
ut_assertIsGreater
ut_assertIsLess
ut_assertIsTrue
ut_assertIsFalse
ut_assertIsNull
ut_assertIsNotNull
ut_assertIsEmpty
ut_assertIsNotEmpty
ut_assertError


And set of functions to retrieve the test results immediately:


ut_GetFailureCount

ut_GetFailureTestName
ut_GetFailureAssertType
ut_GetFailureDescription
ut_GetFailureComment


The test functions should be prefixed with test_, with two specific cases:


test_SetUp - launched before each test
test_TearDown - launched after each test


Confused? What does it mean? What is it good for?

Let's have a look at example in my next post.


Petr

[Petr Schreiber]

The unit testing is concept coming from TDD, and it allows you to test repeatedly your set of functions and check, whether it does not start to fall apart after few modifications.

One of the key ideas of TDD is to write failing tests first, and try to fix the issue one by one, finally reaching fully reliable code (in ideal situation).

Example
Let's say we want to write factorial function on our own (note: thinBasic Math module already provides one)

Factorial can be calculated as following:
n! = (n) * (n-1) * (n - 2) * ... * 1

It is known to be valid only for positive integeres, for zero it returns 1.

To indicate wrong input somehow, let's agree it will return -1 for such a case.

Writing the test
So, we will write the test of the possible user cases first. It is good practice (and required by attached unit) to prefix the test functions with test_.

You can see how the following code manages the testing engine.


Uses "Console"

' -- Include file for unit testing support
#INCLUDE "unitTesting.tBasicU"

' -- Main body of our testing engine
Function TBMain()

Print "Testing factorial..."
' -- Initialize testing engine
ut_Initialize()

' -- Automatically launch all possible tests
ut_LaunchTests()

' -- Report failures found in report, if any
Long nFailures = ut_GetFailureCount()
Long i

If nFailures Then
PrintL
PrintL

For i = 1 To nFailures
PrintL "#"+i
PrintL "Test: " + ut_GetFailureTestName(i)
PrintL "Assert: " + ut_GetFailureAssertType(i)
PrintL "Description: " + ut_GetFailureDescription(i)
PrintL "Comment: " + ut_GetFailureComment(i)
PrintL
Next
Else
PrintL "PASSED"
End If

' -- Save results to default file
ut_SaveLog()

' -- Release engine
ut_Release()

WaitKey

End Function

' -- Our function
Function Fact (ByVal aInput As Long) As Quad

Return 0

End Function

' -- Custom tests
Function test_Fact_NegativeInput()

ut_assertEqual( -1, fact(-1), "Test of factorial for negative value")

End Function

Function test_Fact_For0()

ut_assertEqual( 1, fact(0) , "Test of factorial for input equal to zero")

End Function

Function test_Fact_For1()

ut_assertEqual( 1, fact(1) , "Test of factorial for input equal to one")

End Function

Function test_Fact_ForGeneralPositiveValue()

ut_assertEqual(120, fact(5) , "Test of factorial for one of the possible positive values")

End Function

Function test_Fact_ForTooBigValue()

ut_assertEqual( -1, fact(21), "Test of factorial for value out of bounds, on the positive side")

End Function


The validation of returned values is handled via so called asserts. They indicate expected value and allow to add commentary. In this case, the following line:


ut_assertEqual(120, fact(5) , "fact(5)")

...means: The value of 120 is expected result of fact(5) call. The "fact(5)" is just our commentary to be used as clue in case of failing test.

If you run the code above, you will get the following flood of errors:


#1
Test: TEST_FACT_NEGATIVEINPUT
Assert: UT_ASSERTEQUAL
Description: Expected value=-1, found value=0
Comment: Test of factorial for negative value

#2
Test: TEST_FACT_FORTOOBIGVALUE
Assert: UT_ASSERTEQUAL
Description: Expected value=-1, found value=0
Comment: Test of factorial for value out of bounds, on the positive side

#3
Test: TEST_FACT_FORGENERALPOSITIVEVALUE
Assert: UT_ASSERTEQUAL
Description: Expected value=120, found value=0
Comment: Test of factorial for one of the possible positive values

#4
Test: TEST_FACT_FOR0
Assert: UT_ASSERTEQUAL
Description: Expected value=1, found value=0
Comment: Test of factorial for input equal to zero

#5
Test: TEST_FACT_FOR1
Assert: UT_ASSERTEQUAL
Description: Expected value=1, found value=0
Comment: Test of factorial for input equal to one


Note: The test test_Fact_ForTooBigValue was designed with limitations of QUAD(Int64) in mind. It is well known fact you can't go over 20! with it, because of overflow.

Building it up
Once we have a set of failing tests, we can start to fix the errors. Let's start with input validation.
Thanks to using QUAD as datatype in this model example, we know the input must not be bigger than 20 and also that factorial for negative integers is not defined.

Let's improve the Fact function this way:


Function Fact (ByVal aInput As Long) As Quad

If Not Inside(aInput, 0, 20) Then
Return -1
End If

End Function


When we run the test again:


#1
Test: TEST_FACT_FORGENERALPOSITIVEVALUE
Assert: UT_ASSERTEQUAL
Description: Expected value=120, found value=0
Comment: Test of factorial for one of the possible positive values

#2
Test: TEST_FACT_FOR0
Assert: UT_ASSERTEQUAL
Description: Expected value=1, found value=0
Comment: Test of factorial for input equal to zero

#3
Test: TEST_FACT_FOR1
Assert: UT_ASSERTEQUAL
Description: Expected value=1, found value=0
Comment: Test of factorial for input equal to one

...we can see we fixed two issues at once - test_Fact_NegativeInput and test_Fact_ForTooBigValue are gone. Hurray!

Next step
Then we can get rid of the special case of 0, by following modification:


Function Fact (ByVal aInput As Long) As Quad

If Not Inside(aInput, 0, 20) Then
Return -1
End If

If (aInput = 0) Then
Return 1
End If

End Function


When we run the test, we can see test_Fact_For0 is really gone:


#1
Test: TEST_FACT_FORGENERALPOSITIVEVALUE
Assert: UT_ASSERTEQUAL
Description: Expected value=120, found value=0
Comment: Test of factorial for one of the possible positive values

#2
Test: TEST_FACT_FOR1
Assert: UT_ASSERTEQUAL
Description: Expected value=1, found value=0
Comment: Test of factorial for input equal to one


Finish it!
Well, what about making it actually calculating the factorial?
Let's mod it to this:


Function Fact (ByVal aInput As Long) As Quad

If Not Inside(aInput, 0, 20) Then
Return -1
End If

If (aInput = 0) Then
Return 1
End If

Long i
Quad result = 1

For i = 2 To aInput
result *= i
Next

Return result

End Function


Dramatic, isn't it. Let's run the test:


Testing factorial...PASSED


Oh yeah! So now we can be sure our Fact function works correctly for input in range 1..20.
You don't have to be afraid to rewrite function Fact in other way, because in case of doubt, you can always automatically check it over and over again via the automated test.


Petr

P.S. The 3 phases of implementation are attached below, get the unitTesting.tBasicU from the first post

[RobbeK]

A great tool , Petr .. super !

But in case "ut_assertEqual", is there something as " Equal within tollerance " , or can you set up such a construction ?
Sometimes one starts with well known values without a well defined function , while knowing where it is defined and the knowledge of certain exact points of it, the calculated function in the end (while usable) may fail on exactly those points where you started from.
(Also considering the fact p.e. that for floating points the output (let's consider singles) from oxygen is not the same as TB. some digits at the tail may vary , which make them close but not equal ).

best, Rob

[mike lobanovsky]

@Petr

Yes, unit testing is a great method for keeping projects consistent. I'm not sure if it's feasible to actually preface a project with a set of unit tests but once the project's core is set up, it is reasonable to get into the habit of running your unit tests whenever you add a new feature.

As your project grows, however, the unit test code base becomes so huge (large projects even require a dedicated unit test programmer/maintainer/tester) that its output can hardly fit into a reasonable console buffer. OTOH scrollable list views don't permit individual word coloration. I'd suggest creating a helper GUI app with a scrollable richedit control where you can colorize and otherwise capitalize words however you wish. The richedit's text can also be streamed into a rich format text file.

I have my own in-house implementation of a simple unit test system for FBSL's BASIC, for not only math but also data type casting, flow control, boolean logic, function parameter passing and function returns. The latter two are needed to keep handling of FBSL's Variant variables intact. Dynamic C has its own set of unit tests inherited from TCC. Dynamic Assembler is robust and modifications are seldom.

@Rob

What you call "tolerance" is usually referred to as "epsilon" in programming. And you are correct, epsilon is a must in FPU unit testing, especially for floating point evaluation of Singles at their 6th decimal place; Doubles, at decimal places after the 13th; and Extended floating-point values, at decimal places after the 15th.

FPU precision is also largely language implementation dependent.

As Eros said, all intermediate calc is done in TB via Extended quantities so the difference you're seeing between O2 and TB Singles is due to rounding errors on TB's side. Yes, gentlemen, these are TB's errors because O2 calculates its singles classically, i.e. directly on the FPU, as 32-bit values (DWORD-long values in asm terms) that don't accumulate intermediate calc errors in their least significant digits but rather round them up or down on the fly. There is a similar issue in FBSL where all FPU calc is historically done via Doubles. I wouldn't however call it a big problem in either (general-purpose) language if only one is aware of epsilon and allows for it in one's applied programming practice.

[Petr Schreiber]

Hi guys,

thanks for the feedback!

I think improved asserts comparing floats are good idea, maybe it could be also specified as "check for n-decimals" and so on. Something to think about, thanks!

Mike - the output to console is secondary. The complete log is saved to file, so it can be read in any kind of text viewer. Of course, some advanced UI, like NUnit has, would be nice.
Instead of rich text, I am considering XML. It would be structured way interpretable by custom GUIs.

As for unit test becoming huge and complex - I am fine with this as long as I can sleep peacefully knowing the software does what it does. I think my friend Honza does it this way from project beginning.
When designed well, it is just about adding new tests to existing pool, that is no problem.

I will try to push this as far as possible via reflection, to have somehow user-tested approach, which could be used as base for integrated unit testing in ThinBASIC 2.x.


Petr

P.S. Discussion regarding precision is very interesting to me, but please open new thread for it. I would like to see failing examples, if possible.

[Charles Pegge]

Unit Test: is it watertight? :D


http://4.bp.blogspot.com/_f98opUNuVXc/Si4lFtEZBAI/AAAAAAAAHuE/w1yr5Y43Yng/s400/rhino.jpg

[John Spikowski]

I think he is trying to mount the honey in the red dress. :twisted:

[Petr Schreiber]

Hehe,

okay, adding ut_assertRhino :D


Petr

[Petr Schreiber]

Updated the header to version 1.3


Added support for optional setup routine for each group of tests launched via ut_LaunchTests


In case a function named test_setup() or <prefix>_test_setup() is found, it is launched before each test.


Petr

[Petr Schreiber]

Updated the header (and examples) to version 1.4

Added ut_assertEqualEpsilon
Added ut_assertEqualRounded
-
Added ut_GetFailureCount
Added ut_GetFailureTestName
Added ut_GetFailureAssertType
Added ut_GetFailureDescription
Added ut_GetFailureComment
-
ut_GetLastError (returns string of nonzero length in case you use ut_ functions inproperly)
-
Added support for test_TearDown function, launched after each test
-
Added ut_SaveLog to enable explicit saving
Changed log format to structured form:



<header>
<version>1.4</version>
<date>01-17-2014</date>
<time>12:37:44</time>
<failCount>5</failCount>
</header>
<body>
<fail>
<id>1</id>
<testName>TEST_FACT_NEGATIVEINPUT</testName>
<assertType>UT_ASSERTEQUAL</assertType>
<description>Expected value=-1, found value=0</description>
<comment>Test of factorial for negative value</comment>
</fail>
<fail>
<id>2</id>
<testName>TEST_FACT_FORTOOBIGVALUE</testName>
<assertType>UT_ASSERTEQUAL</assertType>
<description>Expected value=-1, found value=0</description>
<comment>Test of factorial for value out of bounds, on the positive side</comment>
</fail>
<fail>
<id>3</id>
<testName>TEST_FACT_FORGENERALPOSITIVEVALUE</testName>
<assertType>UT_ASSERTEQUAL</assertType>
<description>Expected value=120, found value=0</description>
<comment>Test of factorial for one of the possible positive values</comment>
</fail>
<fail>
<id>4</id>
<testName>TEST_FACT_FOR0</testName>
<assertType>UT_ASSERTEQUAL</assertType>
<description>Expected value=1, found value=0</description>
<comment>Test of factorial for input equal to zero</comment>
</fail>
<fail>
<id>5</id>
<testName>TEST_FACT_FOR1</testName>
<assertType>UT_ASSERTEQUAL</assertType>
<description>Expected value=1, found value=0</description>
<comment>Test of factorial for input equal to one</comment>
</fail>
</body>



Petr

[RobbeK]

Thanks, Rob

[RobbeK]

Petr (and all),
This all gave me an idea , it makes use of the dynamic character of JIT compiling.

In this example, we want to find (just a very simple example) the Golden Ratio value -- but we forgot how to calculate the roots of a parabola.
We let TB sample the thing ( we know (cheating a little) that a DIN A4 etc.. using this ratio , so we only start looking between 0 and 3. (we know it is around 3/2)
TB samples till you agree with accurancy (epsilon) , or TB finds no deeper solution -- in both cases you answer with something else than "y".
TB sets up a DLL now with a function GoldenRatio(x) ...

That's it -- simple as can be.
(needs RTL32.INC)

best Rob