numberFormat question
Mark Schonewille
m.schonewille at economy-x-talk.com
Mon Jun 16 16:15:40 EDT 2014
David,
The numberformat doesn't affect precision. It only affects output as is
shown by
on mouseUp
set the numberformat to "00"
put 0.01*1 into x
set the numberformat to "00.00"
put x*1
end mouseUp
It even only affects the current handler and doesn't affect precision in
calling handlers:
on mouseUp
// default is "0.######"
put a1() into x
put x*1
end mouseUp
function a1
set the numberformat to "00"
return 0.01*1
end a1
The above script returns 0.01 instead of 00.
I don't think LiveCode allows you to adjust the precision as other
languages to with e.g. signed and unsigned integers and floating
precision. You'd have to write special routines for this.
I think there is quite a lot of discussion to find in the archives about
high precision and floating point numbers.
--
Best regards,
Mark Schonewille
Economy-x-Talk Consulting and Software Engineering
Homepage: http://economy-x-talk.com
Twitter: http://twitter.com/xtalkprogrammer
KvK: 50277553
Installer Maker for LiveCode:
http://qery.us/468
Buy my new book "Programming LiveCode for the Real Beginner"
http://qery.us/3fi
LiveCode on Facebook:
https://www.facebook.com/groups/runrev/
On 6/16/2014 20:13, dfepstein at comcast.net wrote:
>
>
> I am trying to set the numberFormat so that calculation is precise enough for the situation. Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary.
> But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101.
> The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold.
> Can anybody see what is going wrong?
>
>
>
> Many thanks.
>
>
>
> David Epstein
>
>
>
> function preciseEnough m,n,k
> -- return the value m + k*n
> -- default numberFormat shows up to 6 decimal places
> -- If m, n, or k has more than that precision, this function sets the numberFormat with a margin of safety
> -- before returning the answer
> put length(m) - offset(".",m) into aPlaces
> put length(n) - offset(".",n) into bPlaces
> put length(k) - offset(".",k) into cPlaces
> put the numberFormat into myString
> if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
> get myString & "####"
> set the numberFormat to it
> end if
> put the numberFormat into nf
> put m + k*n into hold
> return hold
> end preciseEnough
More information about the use-livecode
mailing list