numberFormat question

[J. Landman Gay]

On 6/16/2014, 1:13 PM, dfepstein at comcast.net wrote:
>
>
> I am trying to set the numberFormat so that calculation is precise enough for the situation.  Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary.
> But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101.
> The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold.
> Can anybody see what is going wrong?
>
>
>
> Many thanks.
>
>
>
> David Epstein
>
>
>
> function preciseEnough m,n,k
>     -- return the value m + k*n
>     -- default numberFormat shows up to 6 decimal places
>     -- If m, n, or k  has more than that precision, this function sets the numberFormat with a margin of safety
>     -- before returning the answer
>     put length(m) - offset(".",m) into aPlaces
>     put length(n) - offset(".",n) into bPlaces
>     put length(k) - offset(".",k) into cPlaces
>     put the numberFormat into myString
>     if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
>        get myString & "####"
>        set the numberFormat to it
>     end if
>     put the numberFormat into nf
>     put m + k*n into hold
>     return hold
> end preciseEnough

As Mark said, numberformat is only retained during the local handler so 
you need to reset it each time the handler runs.

When you use #, it means to include a numerical value in that postion 
only if there is an actual value there; if there is no value then that 
position is ignored. If you use 0 in the numberformat instead, empty 
positions are padded with zeros. So, to get the precision you want, you 
don't need to calculate the number of places/positions, just use the # 
as you are now. Include enough #s to cover your longest anticipated 
number, up to LC limits (I think that's 16 places.)

Since numberformat only affects the display, you need to force the 
calculation from a numerical value to a text value. You can do that by 
simply putting the result of the calculation into a field. If you want 
it in a variable, you can use value() to do that instead.

So your whole handler can be like this:

function preciseEnough m,n,k
   put m + k*n into hold
   set the numberformat to "0.##############" -- add more if you want
   put value(hold) into hold
   return hold
end preciseEnough

-- 
Jacqueline Landman Gay         |     jacque at hyperactivesw.com
HyperActive Software           |     http://www.hyperactivesw.com