numerical bug -- a better example

[Ivers, Doug E]

 try this:

on test1
  set the numberFormat to "0.#################"
  put 25000000/.99 into zNumber
  put 10^-4 into roundFactor
  put round(zNumber*roundFactor)/roundFactor into zNumber
  put zNumber
end test1

...yields the expected result.  However:

on test2
  set the numberFormat to "0.#################"
  put 25000000/.99 into zNumber
  put 10^-5 into roundFactor
  put round(zNumber*roundFactor)/roundFactor into zNumber
  put zNumber
end test2

...yields extraneous digits at the end.

I'm running 1.1.1 on Mac OS X.

-----Original Message-----
From: Ivers, Doug E
To: use-revolution at lists.runrev.com
Sent: 3/30/2002 10:30 AM
Subject: numerical bug

Please be aware that  put 10^-1 

yields 

0.10000000000000000555 

There is a similar problem with 10^-2 and 10^-3. 

This is a bug.  I'll code a workaround for now and assume that this will
be fixed in the next release of Rev. 


-- D 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.runrev.com/pipermail/use-livecode/attachments/20020331/6e94f212/attachment.html>