Merge and unicode

[J. Landman Gay]

On 9/9/19 2:39 PM, Paul Dupuis via use-livecode wrote:
> On 9/9/2019 2:13 PM, J. Landman Gay via use-livecode wrote:
>> On 9/9/19 1:08 PM, J. Landman Gay via use-livecode wrote:
>>> It seems that the merge command doesn't respect unicode. Does anyone 
>>> have a workaround? The text I'm inserting is already decoded to UTF16.
>>>
>>
>> I misspoke, sorry. It's the metadata that doesn't respect unicode.
>>
> 
> Can you clarify what you mean when you say the "metadata" doesn't 
> respect Unicode?
> 
> I'm in the middle of a big Unicode problem and have found and reported a 
> ton of bugs where Unicode is not yet everywhere. I'm keenly interested 
> in any I don't know about.

Actually I just double-checked and both merge and metadata may be wrong.

I get UTF8 text from a server that is then textDecoded to UTF16, and an 
html template that I merge with parts of the UTF16 text. In the variable 
watcher, the merged template looks correct but when a field is set to 
the htmltext the result is wrong, diacriticals and curly quotes are 
question marks. My solution for that was to urlEncode the content before 
merging, and urlDecoding when extracting it for display. That works.

In another part of the app I use the same (UTF16) text to set the 
metadata of a line in a field. When the script gets the metadata later, 
diacriticals and curly quotes are strange characters with very high UTF 
numbers.

-- 
Jacqueline Landman Gay         |     jacque at hyperactivesw.com
HyperActive Software           |     http://www.hyperactivesw.com