Monte Goulding's day off? ; -) mergJSON - Documented Bug or Feature?

[Mark Waddingham]

On 2016-06-30 00:06, Monte Goulding wrote:
> I’m sorry mergJSON isn’t working out for you. It sounds to me like you
> actually want to represent your numbers as strings in the JSON. In
> which case mergJSON does have the facility to force things that look
> like numbers to be strings and things that look like sequentially
> indexed arrays to be unordered objects.

This relates to the generation of JSON, I believe?

I think the issue here is the reading of the JSON.

> LiveCode will format numbers using the current numberFormat when the
> user wants to use them as a string. This would probably be a
> reasonable change to the external too. mergJSON is open source so if
> anyone is so inclined they could use EvalExpr to get the numberFormat
> then do something like what the engine does in MCU_r8tos to format the
> string. The repository is at
> https://github.com/montegoulding/mergJson. We might want an extra
> boolean parameter governing whether to apply the numberFormat also I
> think.

I do wonder here whether the problem is that from LiveCode Script's 
point of view (and Lagi's use-case, probably others too) that it would 
be preferable that all non-object/array/string values in the JSON are 
returned as they are in the input file; string values being unescaped 
appropriately (the quotes removed, obviously). This means that the 
LiveCode engine would then do the conversions as appropriate, during 
use.

In particular, this would mean the original representation of numbers as 
they are in the JSON would be preserved - i.e. you would get 1.40 if the 
value in the JSON was 1.40 - not the result of going string -> double -> 
string again.

Unfortunately, from what I can see in the jannson source (the library 
used my mergJSON to read the JSON), there is no mode that allows this.

As LiveCode Script can't distinguish between 
numbers/strings/null/true/false - all are represented by strings, this 
would actually be better than the current situation for many use-cases.

However, it is important to consider the fact that there are use-cases 
where you *do* need to distinguish between a string "1.40" in the JSON 
input and a number 1.40; as well as a string "true" and the boolean 
value true in the JSON input. This is a similar problem to supporting 
'NULL' in database columns - LiveCode Script doesn't distinguish between 
the empty string "" and no value at all (NULL) - so you have to use a 
sideline call to determine if a database field is actually NULL or not.

Prior to LiveCode 7, all values were either strings or arrays - however, 
now they can actually be nothing, booleans, strings, numbers, binary 
strings or arrays and there are operators to determine what a value's 
type 'really' is the 'is strictly' operators. So you can do things like:

     put "1.40" into tNumString -- tNumString is strictly a string
     put tNumString + 1.6 into tNumber -- tNumber is strictly a number

These could provide a basis for providing a JSON importer which allows 
both modes of use:

   1) The 'simpler' everything is a string or an array view.

   2) The 'faithful' things have a specific type view, where you can use 
the 'is strictly' operators to determine the *actual* type of the value 
which was specified in the JSON input. e.g.
       {
          "foo" : true
       }
      Would return an array with key "foo" mapping to 'string' "true"; 
but if you test the value with 'is strictly a boolean' it would return 
true; if you test with 'is strictly a string' it would return false.

There's still a couple of caveats here...

LiveCode Script (as yet) does not have 'proper lists' - at the moment 
'arrays' are used to represent both JSON objects and JSON arrays (JSON 
arrays are sequences of values - lists in our parlance) - so you cannot 
distinguish between something which is an object in the JSON input and 
something which is an array in the JSON input.

I'm not sure yet how we'd make it possible to distinguish between the 
number 1.40 and the string "1.40" in the JSON input but still make it so 
that you 'see' what the precise input (1.40, in this case) was when the 
number is converted to a string. Maybe one option would be to have a new 
internal datatype StringyNumber which would be a String, but act like a 
Number - basically a pair (string, number) with the invariant that 
stringtonum(string) == number (but not the other way around - as that's 
ambiguous). Such a beast would return 'false' for is 'strictly a 
string', but 'true' for is 'strictly a number'. Of course, the problem 
then is that you get an inconsistency with regards things such as 'the 
numberFormat'. Whenever the engine formats a number as a string, it uses 
the numberFormat to work out how to generate the string representation 
of the number. With StringyNumbers, though you get an inconsistency:

     put 3.14159 into tStringyNumber
     set the numberFormat to "#.##"
     put "Number: " & tStringyNumber
     -- as tStringNumber is strictly a number you'd expect it to obey 
numberFormat and so the output would be:
     --    Number: 3.14
     -- However, the point of a stringy-number is to preserve the 
original string rep, so with the currently proposed logic
     -- you would get:
     --    Number: 3.14159

This means the 'naive' approach suggested above is perhaps not quite 
right as it makes the number/string 'fragile' in the sense that you can 
get different outputs depending on what the inputs were (and have no way 
to detect that difference will occur). Therefore, I wonder if we can 
make the numberFormat logic 'work' on strings, and not numeric 
representations... i.e. The operation ConvertToString(StringyNumber -> 
String), should use the string part of the StringyNumber together with 
the numberFormat part to generate the output String - essentially if the 
numberFormat *requires* more trailing decimal places than the string 
part of the stringy-number has, then it should extend the string with 
those zeros to match; if the numberFormat *forces* less decimal places 
than the string part of the stringy-number has then it should format 
from the number part of the stringy-number, ignoring the string part 
altogether (the other option would be to 'round' the string directly, I 
think these two these would be equivalent - but a little bit of maths is 
needed to see which is the most appropriate to minimise error).

Anyway, upshot is, mergJSON uses a library which perhaps does not do 
things 'quite the best way' for integration with a stringy language like 
LiveCode Script (hence the contention Lagi is having); the recent 
changes to LiveCode (since 7) could mean we can have the best of both 
worlds in the future, although there is a bit of core engine work to do 
to make that possible.

Warmest Regards,

Mark.

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps