Binary Floating Point Values (was MOD function...)

Dar Scott dsc at swcp.com
Fri Mar 29 14:39:01 EST 2002


On Friday, March 29, 2002, at 06:00 AM, Michael D Mays wrote:

> Decimal 0.1 when converted to binary is actually slightly greater than
> decimal 0.1. Set the numberFormat to something very precise and 
> you will see
> that .1 is really
>  0.10000000000000000555111512312578

I'm am very pleased to be on this list; I would have never guessed 
that the engine used floating-point binary underneath.  This 
behavior seems to apply to both fractional numbers and integer 
numbers.

I'd rather not have to explain this to the neighbor children.  And 
for now, even my own kids.  I need to figure out some fence within 
which they will be safe.

Dar Scott





More information about the use-livecode mailing list