Strange math behaviour... could someone explain this ?

[Ken Ray]

On 10/8/05 9:47 PM, "Richard Gaskin" <ambassador at fourthworld.com> wrote:

>> You might get Rev to introduce a workaround (e.g. by changing the
>> effective meaning of trunc() to be "round at some ;eve; of precision
>> that will do what I *want*, and then truncate", as Mark suggested) - but
>> I fear that only leads to someone, some day in the future open a BZ to
>> say that Rev gets a different answer to the same calculation than 95% of
>> the installed base of  languages).
> 
> Maybe not.  We know, for example, that Excel already accounts for this
> in its own code, correcting for the critical failure of the
> microprocessor designers.  Anyone have time to check if RB, VB, and
> others do too?

Yes, here's the results of all the environments I could find on my
computers:

Revolution 2.6
    put ((36 - 34.2) * 100)                         >> 180
    put trunc(( 36 - 34.2) * 100)                 >> 179
    
SuperCard 4.1.2
    put ((36 - 34.2) * 100)                         >> 180
    put trunc(( 36 - 34.2) * 100)                 >> 179

HyperCard 2.4.1
    put ((36 - 34.2) * 100)                         >> 180
    put trunc(( 36 - 34.2) * 100)                 >> 179

Flash     MX 6.0
    trace((36-34.2) * 100)                          >> 180
    trace(int((36-34.2) * 100))                    >> 179
    
Director MX 9.0
    alert(string((36-34.2) * 100))                >> 180.0000
    alert(string(integer((36-34.2) * 100)))    >> 180
    
REALBasic 2005 R3
    Msgbox(CStr((36-34.2) * 100))                >> 180
    Dim tVal as Integer
      tVal = ((36-34.2) * 100)
      Msgbox(CStr(tVal))                                >> 179
    
Visual Basic 6.0
    MsgBox(CStr((36-34.2) * 100))               >> 180
    MsgBox(CStr(Int((36-34.2) * 100)))        >> 179

AppleScript 1.9.3
    get ((36-34.2) * 100)                            >> 180.0
    set tVal to ((36-34.2) * 100) as integer   >> 180

Excel 2004 Mac
    =(36-34.2) * 100                                    >> 180
    =TRUNC((36-34.2) * 100)                        >> 180

So it seems like Excel, AppleScript and Director keep it at 180, but all the
rest convert it to 179.

> If we find that other high-level systems follow Excel's lead on this
> one, perhaps there's an even stronger argument to asking Rev to account
> for this processor failure (that is, in addition to the answer being wrong).

I'm still trying to figure out how something as simple as ((36-34.2)*100)
can be anything but 180, or how (as Alex put it) "the binary double
precision representation of 34.2 is inexact"... and if it *is*, isn't there
any other form that could represent 34.2 exactly that's not binary double
precision? 

But that's me...

;-)


Ken Ray
Sons of Thunder Software
Web site: http://www.sonsothunder.com/
Email: kray at sonsothunder.com