numberFormat question
Bob Sneidar
bobsneidar at iotecdigital.com
Mon Jun 16 18:07:28 EDT 2014
Also, if you don’t care about display, just set the *precision* to LC max. Numberformat is as others have posted, really a display function. After you are done, set the numberFormat back to default or “0”. I bit myself in the butt because I had a counter that I was using to append a number to a variable name, and it hosed my names.
Bob S
On Jun 16, 2014, at 11:13 , <dfepstein at comcast.net> <dfepstein at comcast.net> wrote:
>
>
> I am trying to set the numberFormat so that calculation is precise enough for the situation. Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary.
> But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101.
> The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold.
> Can anybody see what is going wrong?
>
>
>
> Many thanks.
>
>
>
> David Epstein
>
>
>
> function preciseEnough m,n,k
> -- return the value m + k*n
> -- default numberFormat shows up to 6 decimal places
> -- If m, n, or k has more than that precision, this function sets the numberFormat with a margin of safety
> -- before returning the answer
> put length(m) - offset(".",m) into aPlaces
> put length(n) - offset(".",n) into bPlaces
> put length(k) - offset(".",k) into cPlaces
> put the numberFormat into myString
> if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
> get myString & "####"
> set the numberFormat to it
> end if
> put the numberFormat into nf
> put m + k*n into hold
> return hold
> end preciseEnough
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
More information about the use-livecode
mailing list