Binary Floating Point Values (was MOD function...)
    Dar Scott 
    dsc at swcp.com
       
    Fri Mar 29 09:39:01 EST 2002
    
    
  
On Friday, March 29, 2002, at 06:00 AM, Michael D Mays wrote:
> Decimal 0.1 when converted to binary is actually slightly greater than
> decimal 0.1. Set the numberFormat to something very precise and 
> you will see
> that .1 is really
>  0.10000000000000000555111512312578
I'm am very pleased to be on this list; I would have never guessed 
that the engine used floating-point binary underneath.  This 
behavior seems to apply to both fractional numbers and integer 
numbers.
I'd rather not have to explain this to the neighbor children.  And 
for now, even my own kids.  I need to figure out some fence within 
which they will be safe.
Dar Scott
    
    
More information about the use-livecode
mailing list