numerical bug -- a better example
Dar Scott
dsc at swcp.com
Sun Mar 31 12:33:01 EST 2002
On Sunday, March 31, 2002, at 05:01 AM, Ivers, Doug E wrote:
> on test2
> set the numberFormat to "0.#################"
> put 25000000/.99 into zNumber
> put 10^-5 into roundFactor
> put round(zNumber*roundFactor)/roundFactor into zNumber
> put zNumber
> end test2
>
> ...yields extraneous digits at the end.
I get this:
25300000.0000000037252903
It seems Revolution uses 8 byte binary floating point for
computations. That means you have only 15 digits to work with. I
count 17 #'s above; using a number as a string will likely show the
15-digit limit. You can see it in the intermediate results of your
calculation, if you insert commands to show those.
Every number has a potential error. For example, 2500000000 is
really ( 1 + err) * 2500000000, as handled by Revolution, that is.
This can accumulate with math operation.
Consider the result:
25300000.0000000037252903
^
15th digit
Given the implementation, this seems reasonable.
That error is just a potential error. It depends on how close a
number can be represented by the underlying floating point data
type. In some calculations, you don't see it. Also, even when the
parameters of math functions have representation errors, there is a
chance that a result is very close to what you had expected.
A simple error demo: Set the numberFormat to that above.
Calculate .1 + 0. Result: 0.10000000000000001
Dar Scott
More information about the use-livecode
mailing list