Troubles with inequality
bobsneidar at iotecdigital.com
Thu Feb 5 11:54:15 EST 2015
So here’s what I came up with. Pretty simple.
put 5.226 into myVar
set the numberFormat to "0.00"
add .001 to myVar
put myvar & cr into myString
set the numberFormat to "0.000"
put myvar after myString
Clearly the value in the variable is not being affected at all. This makes me feel a lot better. The moral of this is, always set the numberFormat appropriately prior to calculations of any kind, and don’t use the contents of fields in calculations.
> On Feb 4, 2015, at 15:20 , J. Landman Gay <jacque at hyperactivesw.com> wrote:
> It's obscure, yes, though the dictionary has this:
> "Important! Changing the numberFormat does not automatically change the format of a number that's already in a container. It affects numbers only when they are calculated and then displayed or used as strings. Otherwise, the number retains its full numeric precision."
> The "already in a container" should be more precise though and specify "in a variable". Fields are containers, but they only hold strings.
> On 2/4/2015 5:02 PM, Graham Samuel wrote:
>> Thanks Jacque, a gold mine of information as usual. But it’s pretty obscure, isn’t it? All this started for me because I wanted to test if two numbers were equal, knowing that they probably weren’t exactly equal to the last decimal place but nevertheless were ‘engineering equal’ as it were - say to six places of decimals. So I tried to truncate them to that length and then compare them. More fool me. But now I think I know what should be done, thanks to you - and to everyone else who replied.
>>> On 4 Feb 2015, at 23:44, J. Landman Gay <jacque at hyperactivesw.com> wrote:
>>> On 2/4/2015 3:01 PM, Graham Samuel wrote:
>>>> So really, if I want an uncomplicated string of characters derived
>>>> from a number via setting the numberFormat and then doing a
>>>> calculation, how do I get it?
>>> You can turn it into a string by putting empty after it:
>>> put 1.55555 into tVar -- number
>>> set the numberformat to "0.00"
>>> add 1 to tVar -- still a number
>>> put tVar into fld 1 -- numberformat applied now, field contains a string "2.55"
>>> add 1 to tVar -- still a number, contains 3.55555
>>> put empty after tVar -- numberformat applies, tVar is a string
>>> Jacqueline Landman Gay | jacque at hyperactivesw.com
>>> HyperActive Software | http://www.hyperactivesw.com
>>> use-livecode mailing list
>>> use-livecode at lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> Jacqueline Landman Gay | jacque at hyperactivesw.com
> HyperActive Software | http://www.hyperactivesw.com
More information about the use-livecode