Binary Floating Point Values (was MOD function...)
Dar Scott
dsc at swcp.com
Fri Mar 29 09:39:01 EST 2002
On Friday, March 29, 2002, at 06:00 AM, Michael D Mays wrote:
> Decimal 0.1 when converted to binary is actually slightly greater than
> decimal 0.1. Set the numberFormat to something very precise and
> you will see
> that .1 is really
> 0.10000000000000000555111512312578
I'm am very pleased to be on this list; I would have never guessed
that the engine used floating-point binary underneath. This
behavior seems to apply to both fractional numbers and integer
numbers.
I'd rather not have to explain this to the neighbor children. And
for now, even my own kids. I need to figure out some fence within
which they will be safe.
Dar Scott
More information about the use-livecode
mailing list