Merge and unicode

[J. Landman Gay]

Well, I've made some changes to the code since I started urlEncoding the 
text before merging so I'll check that again. Paul is right that unicode in 
htmltext needs to be in hex, but the numbers I'm getting back are very high 
(8,000+) and render in the field as strange pictographs. Elsewhere where 
there is no merge, curly quotes translate to the named quote or apostrophe 
entities and are correct.

By metadata I mean the LC term (see the dictionary) that allows you to 
attach some text to a field text chunk. The metadata isn't displayed in the 
field but you can use it for anything you want. In my case the field is a 
list of clickable entries in a table of contents, each with its own 
metadata attached that provides a path to the stack and card the entry 
needs to open.

When I use normal LC text as metadata, diacriticals aren't rendered 
correctly (curly quotes become question marks,) the path is therefore 
incorrect and the click goes nowhere.

Since LC is supposed to be unicode throughout, I'd expect metadata to be 
compatible. The same text appears correctly when not used as metadata.
--
Jacqueline Landman Gay | jacque at hyperactivesw.com
HyperActive Software | http://www.hyperactivesw.com
On September 9, 2019 7:25:28 PM Dar Scott Consulting via use-livecode 
<use-livecode at lists.runrev.com> wrote:

> I think you are trying to think too much about the LC implementation of 
> text. Maybe.
>
> Text in LC is an abstraction of a sequence of code points. Whether it is 
> UTF16 or not is hidden to me. (mostly)
>
> So,
>
> get textDecode( binaryFromServer, "UTF-8" )
>
> should put that into the correct form, if it is really UTF-8.
>
> A data (binary bytes) is interpreted as native encoding if one tries to use 
> it as text. I recommend against this. I try to always textDecode() 
> everything coming in, but I make exceptions at times for ASCII.
>
> I'm not sure what you mean by metadata. Are you referring to HTTP content-type?
>
> Sorry, if I am off on a bunny trail...
>
> Dar
>
>> On Sep 9, 2019, at 4:38 PM, J. Landman Gay via use-livecode 
>> <use-livecode at lists.runrev.com> wrote:
>>
>> It's UTF8 text from a server, which I textDecode to UTF16. When I use the 
>> UTF16 text in a merge, diacriticals and/or curly quotes get mangled. (Same 
>> with setting metadata on field text too.)
>>
>> On 9/9/19 4:16 PM, Dar Scott Consulting via use-livecode wrote:
>>> I'm not sure I understand.
>>> Do you mean "encoded to UTF-16"? In that case you should decode that to 
>>> convert it to internal text. And then try merge. (Which still might have 
>>> problems, I suppose.)
>>>> On Sep 9, 2019, at 12:08 PM, J. Landman Gay via use-livecode 
>>>> <use-livecode at lists.runrev.com> wrote:
>>>>
>>>>
>>>> It seems that the merge command doesn't respect unicode. Does anyone have a 
>>>> workaround? The text I'm inserting is already decoded to UTF16.
>>>>
>>>>
>>>> --
>>>> Jacqueline Landman Gay         |     jacque at hyperactivesw.com
>>>> HyperActive Software           |     http://www.hyperactivesw.com
>>>>
>>>>
>>>> _______________________________________________
>>>> use-livecode mailing list
>>>> use-livecode at lists.runrev.com
>>>> Please visit this url to subscribe, unsubscribe and manage your 
>>>> subscription preferences:
>>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>> _______________________________________________
>>> use-livecode mailing list
>>> use-livecode at lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your 
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>>
>> --
>> Jacqueline Landman Gay         |     jacque at hyperactivesw.com
>> HyperActive Software           |     http://www.hyperactivesw.com
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your 
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your 
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode