numerical bug -- a better example

Dar Scott dsc at
Sun Mar 31 12:33:01 EST 2002

On Sunday, March 31, 2002, at 05:01 AM, Ivers, Doug E wrote:

> on test2
>   set the numberFormat to "0.#################"
>   put 25000000/.99 into zNumber
>   put 10^-5 into roundFactor
>   put round(zNumber*roundFactor)/roundFactor into zNumber
>   put zNumber
> end test2
> ...yields extraneous digits at the end.

I get this:

It seems Revolution uses 8 byte binary floating point for 
computations.  That means you have only 15 digits to work with.  I 
count 17 #'s above; using a number as a string will likely show the 
15-digit limit.  You can see it in the intermediate results of your 
calculation, if you insert commands to show those.

Every number has a potential error.  For example, 2500000000 is 
really ( 1 + err) * 2500000000, as handled by Revolution, that is.  
This can accumulate with math operation.

Consider the result:
               15th digit

Given the implementation, this seems reasonable.

That error is just a potential error.  It depends on how close a 
number can be represented by the underlying floating point data 
type.  In some calculations, you don't see it.  Also, even when the 
parameters of math functions have representation errors, there is a 
chance that a result is very close to what you had expected.

A simple error demo:  Set the numberFormat to that above.  
Calculate .1 + 0.  Result: 0.10000000000000001

Dar Scott

More information about the use-livecode mailing list