Using Arrays - math precision
Mark Wieder
mwieder at ahsoftware.net
Sat Jun 4 18:01:27 CDT 2011
Jerry-
Saturday, June 4, 2011, 3:52:03 PM, you wrote:
> LC uses IEEE double-precision binary floating point numbers for
> calculations. These are 64 bits: 52 for mantissa, 11 for exponent
> and 1 for sign. So exponent range won't be a problem, neither will 6
> decimal places, as long as the number of significant digits is less
> than 15. Be sure to round to that least digit, because calcs are in
> binary.
Thanks - good to know. But I thought the default numberFormat was to
display only six digits?
--
-Mark Wieder
mwieder at ahsoftware.net
More information about the use-livecode
mailing list