Importing unicode UTF8 text files - followup

[J. Landman Gay]

On 8/19/2015 2:34 PM, Peter TB Brett wrote:
> On 2015-08-19 21:09, J. Landman Gay wrote:
>> Just to follow up: When importing foreign language files, "file" isn't
>> working. Non-ascii characters show up as question marks and textDecode
>> does nothing.
>>
>> Using "binfile" and then textDecoding does import correctly but then
>> text chunking by lines fails. Line endings are imported as 2 bytes
>> using byteToNum 13,10 (in that order) which isn't the line ending
>> standard for any OS. I have to replace those specifically.
>>
>> These are text files saved as UTF-8, created on Mac OS X and imported
>> on Mac OS X. I'm using LC 7.0.6.
>
> Hmm, this all sounds quite problematic.
>
>
> Obvious thing to check: are you sure your files are valid UTF-8?  The
> following command will print "0" if the file is valid UTF-8 and "1"
> otherwise:
>
>      iconv -f UTF-8 your_file > /dev/null ; echo $?
>
>
> Otherwise, if you have a test case that you don't mind sharing, could
> you please file a bug report and add me (e-mail address below) to the Cc
> list?  Otherwise, please e-mail me directly.  If you could test with
> LiveCode 7.0.1-rc-1, that would also be quite helpful.
>
> I *definitely can't* guarantee a quick fix, but it's possible something
> obvious is going wrong or we can find a quick workaround.
>
>                                       Peter

Thanks very much. The terminal command returns 0 (and BBEdit can handle 
the file too) so it seems to be valid. I'll submit a bug report, I can 
share the file without any issues.

-- 
Jacqueline Landman Gay         |     jacque at hyperactivesw.com
HyperActive Software           |     http://www.hyperactivesw.com