Monte Goulding's day off? ; -) mergJSON - Documented Bug or Feature?

Wed Jun 29 13:43:39 EDT 2016

The letter of the JSON spec does make perfect sense and the C library does precisely what it should - it uses the appropriate C types to represent the values which are encoded in the JSON file.

Of course, in C, you have to explicitly convert from numbers to strings and as such *always* have to explicitly think about the number of decimal places you are going to use when displaying numbers.

The issue here is that the external interface the external uses can only return strings to LiveCode and as such (in order to preserve precisely what is in the JSON file) has to format the numeric strings to a degree of precision such that when the engine converts them back to actual numbers internally, they are precisely the same number as was converted from the original string.

The issue here is that 1.4 is not exactly representable as a binary floating point value - the value that is represented is actually 1.4+/-epsilon where epsilon is a very small error value. So when given a number converted from the string "1.4" it could actually have come from a huge range of different numeric strings (all very close together) - it's an entirely arbitrary decision to choose any one of these possibilities when converting back to a string, the one which is chosen is generally the one which is the fastest to generate algorithmically.

If the original C library had been written to interface to a stringy language (like LiveCode) via an external code interface which only understands strings (which is the case with the mergJson external) then it might well have kept the string representation of the number alongside that of the actual number; or indeed only passed around the original string representation. However, it was not designed with that in mind and as such there is some friction between using it from LiveCode in this way.

Fortunately mergJSON is not the only option - there is LCB library providing JSON import and export as well as several LiveCode script JSON parsers around. They all have their pros and cons I'm sure - so pick one which better suits you direct needs.

Warmest Regards,

Mark.

Sent from my iPhone

> On 29 Jun 2016, at 17:57, Lagi Pittas <iphonelagi at gmail.com> wrote:
> 
> Hi darr
> 
> I think the upshot of this conversation is that Monte used a C library for
> the JSON  that followed the JSON documentation to the letter, whether the
> letter made practical sense.
> If this is what Doug Crockford envisaged well there is another God we have
> to stop worshiping ;-).
> 
> update
> 
> Just read through https://www.rfc-editor.org/rfc/rfc7158.txt
> 
> And nowhere can I see  it stipulate  that it has to be read  or written to
> the maximum number of digits of precision, only that the precision can be
> implementation dependent.
> 
> So we can thank (or not)  the creator of the C-library that Monte used for
> the illogical (IMHO) way it works.
> 
> Anyway Darr - it seems your JSON library is needed so we can make our own
> tweaks if need be - but I bet it works the way I envisage it here.
> 
> Regards Lagi
> 
> p.s. Don't get me wrong it's not a problem to do the rounding, but if I
> hand build a JSON file with 1.40 , 1.20, etc in it and I read and then
> output it again I expect the data to remain the same. Same with CSV same
> with a livecode script.
> 
> 
>> On 29 June 2016 at 17:11, Dar Scott <dsc at swcp.com> wrote:
>> 
>> I'm coming in this late and I'm confused.
>> 
>> I made a JSON library a couple years ago (not an external) and I have been
>> encouraged to make it available, but with JSON now being built-in, I didn't
>> bother.  Now, there is MergJSON?
>> 
>> 
>>> On Jun 29, 2016, at 9:06 AM, Mark Waddingham <mark at livecode.com> wrote:
>>> 
>>> On 2016-06-29 16:58, Lagi Pittas wrote:
>>>> Hi Mark,
>>>> Thanks for the replies. But your second response I don't understand and
>> the
>>>> whole point of what I was getting at.
>>>> "I suppose there could be an option to choose a 'number of decimal
>> places'
>>>> option passed to the library - however, you will then potentially lose
>>>> information in your numbers. Therefore, it is probably better to do the
>>>> rounding in script after loading the JSON file."
>>>> If  "reals" are returned as  strings anyway then if the real says 2.30
>> how
>>>> can I be losing any precision if 2.30 is what's in the file?. The option
>>>> though would be a good idea but in the calling routine to MergJSON.. If
>> it
>>>> is left blank or false then it uses the current setting of number format
>>>> or if there is a string then that will be the format it uses.
>>> 
>>> Ah - this is the critical point...
>>> 
>>> MergJSON uses a C library which converts the text of a JSON input into a
>> tree of values - these values are arrays, objects, numbers, strings,
>> booleans etc. i.e. The output of this library is then processed by the
>> external to convert it to a LiveCode array and string values...
>>> 
>>> This means that the external gets numbers from the JSON as IEEE754
>> doubles, and then has to format them for return as strings. As the original
>> string has already been converted to a number at this point, the external
>> must format the numbers with enough decimal places so that you don't lose
>> any information. Hence why you are probably best off processing the output
>> of the external in LiveCode Script, rounding any numbers you know need to
>> be rounded.
>>> 
>>>> If I look at the Arrays using the variable display in the debugger they
>> are
>>>> all showing as 2 decimals so I would expect them to export in the same
>>>> format - or am I again missing something about the debugger?
>>> 
>>>> If added  global variable xx in the debugger with value 1.3456 and that
>> is
>>>> how it shows and ll 2 decimal place prices still show the same so LC
>>>> "knows" how I want to see them.
>>> 
>>> The debugger displayed values using the numberFormat settings - usually
>> 6 decimal places.
>>> 
>>> However, an important thing to remember about LiveCode Script is that if
>> something is a string, then it will remain a string. You only get actually
>> numbers internally when you perform operations on values which need to be
>> numbers. So:
>>> 
>>>   put 1.43858239402803492304
>>> 
>>> will display:
>>> 
>>>   1.43858239402803492304
>>> 
>>> But:
>>> 
>>>  put 1.43858239402803492304 + 0
>>> 
>>> will display:
>>> 
>>>  1.438582
>>> 
>>> As numbers are rounded using numberFormat when converted to strings.
>>> 
>>> Warmest Regards,
>>> 
>>> Mark.
>>> 
>>> --
>>> Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
>>> LiveCode: Everyone can create apps
>>> 
>>> _______________________________________________
>>> use-livecode mailing list
>>> use-livecode at lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> 
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode