Math issue, isn't it?

Colin Holgate coiin at rcn.com
Mon May 11 12:36:56 EDT 2009


On May 11, 2009, at 12:11 PM, Bob Sneidar wrote:

> I think we are mincing words here. It's a simple matter of changing  
> the way the math function works.

I understand that it's an oddity that the computer doesn't quite see  
how simple the numbers are, but that happens for humans too. Take a  
simple case of expressing 2/3 as a floating point decimal number, you  
can't actually type out the exact value, even with unlimited monkeys  
to do the typing for you. That's a case where decimal has its  
limitations. Computers have different limitation cases, because  
they're working in binary.

That said, having a mathPrecision equivalent to numberFormat would  
solve the issue too, any tiny amounts below the mathPrecision value  
would be rounded up or down.





More information about the use-livecode mailing list