Using Arrays - math precision

[Jerry J]

On Jun 3, 2011, at 11:35 PM, Mark Wieder wrote:

> Van-
> 
> Friday, June 3, 2011, 9:38:47 PM, you wrote:
> 
>> Many thanks to Mike and Mark for helping me get my array and
>> datagrid issues resolved to date.
> 
>> The latest issue is in calculations.  I need to have calculations
>> done in floating numbers and then the array finishes calculating it
>> truncates at the 6th decimal.  I need to go out to at least E-23. 
>> Is there a way of setting this behaviour somewhere?
> 
> Check out the numberformat property. But that's quite an array you've
> got there - I doubt you'll be getting that kind of precision out of
> the numeric library. Untested, though - ymmv.

LC uses IEEE double-precision binary floating point numbers for calculations. These are 64 bits: 52 for mantissa, 11 for exponent and 1 for sign. So exponent range won't be a problem, neither will 6 decimal places, as long as the number of significant digits is less than 15. Be sure to round to that least digit, because calcs are in binary.

--Jerry Jensen