numberFormat question
Kay C Lan
lan.kc.macmail at gmail.com
Mon Jun 16 20:53:55 EDT 2014
On Tue, Jun 17, 2014 at 4:34 AM, J. Landman Gay
<jacque at hyperactivesw.com> wrote:
> set the numberformat to "0.##############" -- add more if you want
I think the reason David isn't doing that, and is trying to limit the
number of # to the absolute minimum is because of this note in the
Dictionary:
Note: Since LiveCode does not use decimal numbers for its internal
calculations (for reasons of speed), the decimal representation of a
number is sometimes slightly off the correct number. For example,
10^-1 is equal to 0.1, but is calculated (to eighteen decimal places)
as 0.100000000000000006. Because of this, setting the numberFormat to
specify many decimal places after the decimal point may produce
unexpected results in a statement that tests for an exact number. To
prevent this, either avoid setting the numberFormat to a value more
precise than you need, or use the abs function instead of the =
operator to test equality:
I think we all know that at least once or twice a year we get a post
to this List questioning why LC is getting it's math wrong and the
reason is as stated above. For anyone whose app requires precision
math that shouldn't be a note, it should be a CAUTION! It's so easy to
have your app work perfectly 99.9999999999999999% of the time... but
then ruin someone's day.
More information about the use-livecode
mailing list