math, bigly

[Mark Wieder]

On 08/09/2017 02:58 AM, Mark Waddingham via use-livecode wrote:

> The engine hasn't ever done range checks on integers - mainly because 
> numbers are doubles so (in most ases) you just lose accuracy / precision 
> as the numbers get larger. In this case it *could* because it is 
> actually using integers.

I just used baseconvert as an example to show the problem. The one that 
caused the problem for me was bitXor. It wasn't a big deal to write a 
string-oriented version that could handle any inputs, but the problem 
for me is more the way the engine parses the data.

Looking into the engine code, by the time you get to the actual bitXor 
code, the input arguments are already unsigned integers. Since I was 
passing 20-hex-digit values to bitXor, I would have expected some kind 
of runtime warning or error or something. It's not just a loss of 
precision here, it's a totally wrong answer with no indication that 
anything has gone astray. Writing a string-oriented function to handle 
this wasn't a problem - realizing that something had gone wrong was.

-- 
  Mark Wieder
  ahsoftware at gmail.com