Merge and unicode

[Paul Dupuis]

Doesn't any Unicode in htmlText of a field need to be in HTML form (i.e. 
#<hexnumber>;?

I thought htmlText turns any non ASCII into either hex encoded html or, 
where html entity names exists, uses html entity names.


On 9/9/2019 6:35 PM, J. Landman Gay via use-livecode wrote:
> On 9/9/19 2:39 PM, Paul Dupuis via use-livecode wrote:
>> On 9/9/2019 2:13 PM, J. Landman Gay via use-livecode wrote:
>>> On 9/9/19 1:08 PM, J. Landman Gay via use-livecode wrote:
>>>> It seems that the merge command doesn't respect unicode. Does 
>>>> anyone have a workaround? The text I'm inserting is already decoded 
>>>> to UTF16.
>>>>
>>>
>>> I misspoke, sorry. It's the metadata that doesn't respect unicode.
>>>
>>
>> Can you clarify what you mean when you say the "metadata" doesn't 
>> respect Unicode?
>>
>> I'm in the middle of a big Unicode problem and have found and 
>> reported a ton of bugs where Unicode is not yet everywhere. I'm 
>> keenly interested in any I don't know about.
>
> Actually I just double-checked and both merge and metadata may be wrong.
>
> I get UTF8 text from a server that is then textDecoded to UTF16, and 
> an html template that I merge with parts of the UTF16 text. In the 
> variable watcher, the merged template looks correct but when a field 
> is set to the htmltext the result is wrong, diacriticals and curly 
> quotes are question marks. My solution for that was to urlEncode the 
> content before merging, and urlDecoding when extracting it for 
> display. That works.
>
> In another part of the app I use the same (UTF16) text to set the 
> metadata of a line in a field. When the script gets the metadata 
> later, diacriticals and curly quotes are strange characters with very 
> high UTF numbers.
>