Using Arrays - math precision

Jerry J jhj at jhj.com
Sat Jun 4 19:21:23 EDT 2011


On Jun 4, 2011, at 4:01 PM, Mark Wieder wrote:

> Jerry-
> 
> Saturday, June 4, 2011, 3:52:03 PM, you wrote:
> 
>> LC uses IEEE double-precision binary floating point numbers for
>> calculations. These are 64 bits: 52 for mantissa, 11 for exponent
>> and 1 for sign. So exponent range won't be a problem, neither will 6
>> decimal places, as long as the number of significant digits is less
>> than 15. Be sure to round to that least digit, because calcs are in
>> binary.
> 
> Thanks - good to know. But I thought the default numberFormat was to
> display only six digits?

I think thats true. Now as for why... ??
--Jerry





More information about the use-livecode mailing list