numberFormat question

[Bob Sneidar]

Also, if you don’t care about display, just set the *precision* to LC max. Numberformat is as others have posted, really a display function. After you are done, set the numberFormat back to default or “0”. I bit myself in the butt because I had a counter that I was using to append a number to a variable name, and it hosed my names.  

Bob S


On Jun 16, 2014, at 11:13 , <dfepstein at comcast.net> <dfepstein at comcast.net> wrote:

> 
> 
> I am trying to set the numberFormat so that calculation is precise enough for the situation.  Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary. 
> But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101. 
> The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold. 
> Can anybody see what is going wrong? 
> 
> 
> 
> Many thanks. 
> 
> 
> 
> David Epstein 
> 
> 
> 
> function preciseEnough m,n,k 
>    -- return the value m + k*n 
>    -- default numberFormat shows up to 6 decimal places 
>    -- If m, n, or k  has more than that precision, this function sets the numberFormat with a margin of safety 
>    -- before returning the answer 
>    put length(m) - offset(".",m) into aPlaces 
>    put length(n) - offset(".",n) into bPlaces 
>    put length(k) - offset(".",k) into cPlaces 
>    put the numberFormat into myString 
>    if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then 
>       get myString & "####" 
>       set the numberFormat to it 
>    end if 
>    put the numberFormat into nf 
>    put m + k*n into hold 
>    return hold 
> end preciseEnough 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode