Problem with Latin 1 (I think)

[Ben Rubinstein]

On 06/02/2012 16:21, Geoff Canyon Rev wrote:
> Okay, so I'm at a loss. This worked, but...why?
>
> I first have to *encode* it, then *decode* it? If it's already UTF8, why am
> I encoding it as UTF8? is it that some of it is encoded, and the encoding
> function is encoding the rest, but knows not to encode the characters that
> are already encoded? If that's the case, why isn't the decode function
> smart enough to only decode the characters that need it? Or am I completely
> misunderstanding?
> ...
>>   put unidecode(uniencode(pContents,"UTF8"),"english")

It is confusing - and unfortunately the unixxcode stuff is still a bit 
underspecified.

Essentially the way to convert between character sets is via Unicode (back in 
the hypercard days I had an external for this job, called "viaUnicode").

In an ideal world there might be a single function "convertCharset", which 
takes some text, a source character encoding and a destination character 
encoding.  For historical reasons, LiveCode instead has two functions, one 
that converts text from (some encoding) into Unicode; and from Unicode into 
(some encoding).  The former is called "uniEncode" (because it encodes text as 
Unicode); the latter "uniDecode".  So Malte's statement above converts 
pContents from UTF8 to Unicode; then passes that as the first parameter to 
uniDecode, to be converted from Unicode to... "english".

It's at that last point that I say "WTF?".  What character encoding is 
"english", FFS?  (I tend to find dealing with character encoding issues makes 
me swear a lot.)

The answer in the curious world of MetaCard was that on a Mac, "english" meant 
the MacRoman character set.  On Windows it meant, I think, Windows Latin 1, 
although it might be ISO-8859-1, which is almost the same but not quite.

However this is really unhelpful, because very often I know that my source 
encoding is, and it may not be related to the platform my app happens to be 
running on right now.  And there are a world of other character encodings out 
there, which I'd like to be able to address directly.  So I and others have 
long hoped that the modern world of LiveCode would be able, in this respect, 
to move beyond the curious world of MetaCard.

Hence...
	http://quality.runrev.com/show_bug.cgi?id=3674


Ben