Text encoding.

[Alex Tweedly]

On 02/09/2021 13:01, David Bovill via use-livecode wrote:
> Thanks for the question Alex, Im wrestling with the same issues - but so far got no responses from encoding gurus here :)
>
> This is my understanding:
>
> 1) Yes its recommended to textEncode text that comes from outside into Livecodes internal native format (which is utf16).  Livecode handles everything internally transparently from then on - which I guess means all usual language and control operations expect this utf16 internal format. My guess is this is why a few things have got slower as compared with early versions of Livecode.


Hi David, thanks for the suggestion. But I think you have (maybe) made 
the same mistake I made.

I blame it on the dictionary entry being truncated in one place and not 
as clear as it could be in others. The correct part to focus on is

> that you explicitly*textEncode* any text you send outside LiveCode and 
> textDecode all text received into LiveCode
>
i.e. you *decode* on the way in to Livecode (confirmed by the examples 
in the dictionary, not by the text description).

So the code I was using, sent earlier  in reply to Mark W's email said 
(something like):

   put URL ("file:" & specialFolderPath("home") & "/warpeace.txt") into 
tText
   ...
   put quote&"!?,.:;[]{}()$%^&<>=+-#*\/" into tList
   ...
    put textencode(pStr, "UTF8") into pStr

i.e.I encoded it on the way in.

It should have been

    put textDecode(pStr, "UTF8") into pStr

With that changed, my tiny test script works properly (and quickly). I 
haven't yet put this back into my actual app script - it's way past my 
bedtime :-), but I now expect it will be OK.

Mark - I hope this helps - and hopefully you can conclude there's no 
error, just some dictionary entries to improve. If (when) I confirm this 
in my actual app, I'll create a bugreport for the dictionary.

Alex.