numberFormat question

[Mark Schonewille]

David,

The numberformat doesn't affect precision. It only affects output as is 
shown by

on mouseUp
    set the numberformat to "00"
    put 0.01*1 into x
    set the numberformat to "00.00"
    put x*1
end mouseUp

It even only affects the current handler and doesn't affect precision in 
calling handlers:

on mouseUp
    // default is "0.######"
    put a1() into x
    put x*1
end mouseUp

function a1
    set the numberformat to "00"
    return 0.01*1
end a1

The above script returns 0.01 instead of 00.

I don't think LiveCode allows you to adjust the precision as other 
languages to with e.g. signed and unsigned integers and floating 
precision. You'd have to write special routines for this.

I think there is quite a lot of discussion to find in the archives about 
high precision and floating point numbers.

--
Best regards,

Mark Schonewille

Economy-x-Talk Consulting and Software Engineering
Homepage: http://economy-x-talk.com
Twitter: http://twitter.com/xtalkprogrammer
KvK: 50277553

Installer Maker for LiveCode:
http://qery.us/468

Buy my new book "Programming LiveCode for the Real Beginner" 
http://qery.us/3fi

LiveCode on Facebook:
https://www.facebook.com/groups/runrev/

On 6/16/2014 20:13, dfepstein at comcast.net wrote:
>
>
> I am trying to set the numberFormat so that calculation is precise enough for the situation.  Since I don't know ahead of time how many decimal places will be used, I wrote a function that I hoped would adjust things as necessary.
> But when this function is called with parameters m = 1.09131 and n = .0000001 and k = 1, it returns 1.09131 rather than 1.0913101.
> The last couple of lines were added for testing, and the debugger shows that the numberFormat is being set correctly but that the truncated value is put into hold.
> Can anybody see what is going wrong?
>
>
>
> Many thanks.
>
>
>
> David Epstein
>
>
>
> function preciseEnough m,n,k
>     -- return the value m + k*n
>     -- default numberFormat shows up to 6 decimal places
>     -- If m, n, or k  has more than that precision, this function sets the numberFormat with a margin of safety
>     -- before returning the answer
>     put length(m) - offset(".",m) into aPlaces
>     put length(n) - offset(".",n) into bPlaces
>     put length(k) - offset(".",k) into cPlaces
>     put the numberFormat into myString
>     if max(aPlaces,bPlaces,cPlaces) + 4 > length(myString) then
>        get myString & "####"
>        set the numberFormat to it
>     end if
>     put the numberFormat into nf
>     put m + k*n into hold
>     return hold
> end preciseEnough