Strange math behaviour... could someone explain this ?

Mark Waddingham 36degrees at runrev.com
Sun Oct 9 08:41:09 EDT 2005


Indeed, I believe MS Excel and most financial applications use a Decimal
or BCD representation for arithmetic (or use floating-point in such a
way to ensure the results are equivalent).

Floating-point arithmetic (which Revolution uses) works entirely in base
2. However, while all decimal integers have an accurate representation
in binary, not all decimal fractions do and this is where the
approximation error comes from.

For example:
 1/8 = 0.125 (decimal) and 0.001 (binary)
But
 1/5 = 0.2 (decimal) and 0.0011001100110011... (binary)
Note that the binary representation of 1/5 is a recurring fraction and
as such admits no finite (non-rational) representation.

In a BCD representation you fix the amount of precision you want after
the decimal point, and fix the maximum value you want to represent. e.g.
  DDDDDDDDDDDDDDDD.DDDDDD
would probably be sufficient for most financial applications. Then you
actually store a string of decimal digits as the representation of your
number and perform calculations on them as you would in high-school. The
downside is that such numbers have a very fixed range, they take up more
space and are a great deal slower to manipulate.

Alternatively, you can use a Decimal representation. This is where,
again, you choose the maximum precision you require after the decimal
point (i.e. n decimal places) and then normalise all numbers by
multiplying them by 10^n so that the fractional part disappears and you
can just manipulate via integer operations. This will effectively give
you the ability to manipulate numbers as large as you want, but the
bigger the number the greater the amount of storage it provides.

One down-side of both of these representations is anything involving
trigonometric or nth-root calculations become exceedingly slow and/or
unsuitably inaccurate.

You can achieve the same effect as a decimal representation (with a
little care) in Revolution by always storing values at the end-points of
your calculations as a string with a fixed number of decimal places. So,
as suggested elsewhere on this thread:
  - set up the numberFormat property appropriately
  - coerce your number to a string by doing something like
      put tNumber & empty into tStringRepresentation
Or, alternatively, use the 'format' function:
  - put format("%.6f", tNumber) into tStringRepresentation
(here the '6' specifies 6 decimal places, but can be any number you
wish)

One point to make about double precision floating point numbers (which
Revolution utilises internally) is that they have an accuracy of at most
15 *significant* digits - rather than decimal places.

What this means is that 
   0.123456789012345
can be accurately represented but
  12.345678901234501
will be 'rounded' to:
  12.345678901234500

Warmest Regards,

Mark.





More information about the use-livecode mailing list