Rounding errors when subtracting

Mark Smith mark at maseurope.net
Fri Jun 20 10:18:29 EDT 2008


Ian, take a look at the 'numberFormat' property in the docs.

set the numberformat to "0.#########"
put 51.905719161 - 51.905658591

produces 0.00006057 - which is correct, I think.

Best,

Mark

On 20 Jun 2008, at 14:38, Ian Wood wrote:
> Help!
>
> Rev Studio 2.9.0 on OS X 10.5.2 Intel
>
> 'put 51.905719161 - 51.905658591'
>
> results in 0.000061 instead of 0.0000607
>
> Is there some global setting I'm missing, or is this a bug (which I  
> can't find on RevQC)?
>
> I'm getting the same error if I divide by 1,000,000. :-(
>
> Ian
> _______________________________________________
> use-revolution mailing list
> use-revolution at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your  
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-revolution




More information about the use-livecode mailing list