Storing json - mergeJSON, textdecode and "\u00a0”

[David Bovill]

An update on my flailing attempts to get a robust way to go Fromm arrays to json and back. Essentially the story is stuff that works on OSX fails now and then on the server - and I’m trying to track down why?

It’s an intermittent problem and hard to pinpoint as the json files have a lot of content -so finding a character that is causing the problem is not easy. My assumption is that this is an encoding issue - but attempts at using textDecode() are not helping.

I found one instance that was causing the array encoding to fail:
> quote_type
> put textDecode (utf8JSON, "UTF-8") into rJSON
> replace "\u00a0” with empty in rJSON

While this is fixing the cases I have so far - I am not sure what the underlying encoding issue is here. Its a unicode non-breaking-space - https://www.fileformat.info/info/unicode/char/00a0/index.htm but I’m not sure about the encoding and whether its valid utf8?

Bit stuck any thoughts?

📆    Schedule a call with me
On 15 Aug 2021, 11:46 +0100, David Bovill <david.bovill at gmail.com>, wrote:
> I’ve been historically a bit lazy in taking care of how I store json data - so mostly it works but some times I get a different result on the server than I do locally on OSX. I suspect this is due to encoding differences in how the data is stored and which characters are in the json. I’d like to standardise on utf8 encoded json as my text storage of choice, and use a standard way to store this while in Livecode I parse the json and just deal with arrays.
>
> As I understand it I should really be using textEncode and textDecode() - and I should store the utf8 encoded json files using “binfile:” not “file:” - however this is not what the dictionary entry fo textEncode() illustrates:
> > quote_type
> > put textEncode(field "output","UTF-8") into tOutput
> > put tOutput into url ("file:output.txt")
>
> So if I want to implement best practice in storing utf8 encoded json locally on Mac, on mobile and on unix servers according to:
> > quote_type
> > The textEncode function takes text, and returns it as binary data, encoded with the specified encoding.
> > It is highly recommended that any time you interface with things outside LiveCode (files, network sockets, processes, etc) that you explicitly textEncode any text you send outside LiveCode and textDecode all text received into LiveCode. If this doesnt happen, a platform-dependent encoding will be used (which normally does not support Unicode text).
>
> How should I do that? Should I not do something like this (in pseudocode):
> > quote_type
> > function fetchFromUrl jsonURL
> >   put url someURL into utf8JSON
> >   put textDecode(jsonURL, "UTF-8") into encodedJSON
> >   put json_ToArray (encodedJSON) into someArray
> >   return someArray
> > end fetchFromUrl
> >
> > command exportToUrl someArray, jsonURL
> >   put json_FromArray (someArray) into encodedJSON
> >   put textEncode(encodedJSON, "UTF-8") into decodedJSON
> >   put decodedJSON into url someURL
> > end exportToUrl
>
> And. If so - should I be using “binfile:” or “file:” for writing to the file system?
>
> 📆    Schedule a call with me