Using Arrays - math precision
jhj at jhj.com
Sat Jun 4 17:52:03 CDT 2011
On Jun 3, 2011, at 11:35 PM, Mark Wieder wrote:
> Friday, June 3, 2011, 9:38:47 PM, you wrote:
>> Many thanks to Mike and Mark for helping me get my array and
>> datagrid issues resolved to date.
>> The latest issue is in calculations. I need to have calculations
>> done in floating numbers and then the array finishes calculating it
>> truncates at the 6th decimal. I need to go out to at least E-23.
>> Is there a way of setting this behaviour somewhere?
> Check out the numberformat property. But that's quite an array you've
> got there - I doubt you'll be getting that kind of precision out of
> the numeric library. Untested, though - ymmv.
LC uses IEEE double-precision binary floating point numbers for calculations. These are 64 bits: 52 for mantissa, 11 for exponent and 1 for sign. So exponent range won't be a problem, neither will 6 decimal places, as long as the number of significant digits is less than 15. Be sure to round to that least digit, because calcs are in binary.
More information about the use-livecode