Unicode with answer file and manipulating text fields

[Trevor DeVore]

On Thursday, September 18, 2003, at 12:53  PM, Tuviah Snyder wrote:
> Problem here is that one of your unicode characters may contain the 
> newline
> character and therefore Rev is returning the wrong line offsets 
> (because the
> chunk stuff is still working in ASCII mode). Support for unicode 
> chunks is
> planned in the near future but you can look at 'Unicode Text 
> Manipulation
> Samples' in the developers contributions library for sample scripts to 
> roll
> your own unicode compatible parsing routines.
>
> I would also suggest for this sort of things getting the unicodetext 
> of a
> field, modifying it, and then resetting it.

Thanks Tuviah.  I downloaded the stack and using the functions from the 
stack now have the text inserting on the correct lines:

put uniEncode (tNewMovieName & tab & pMoviePath, "Japanese") into 
tNewLine
put unicodeText of field "movieList" into tMovieList
put CountUnicodeLines (tMovieList) + 1 into tNextLine
if (tNextLine > 1) then put uniEncode (return, "Japanese") after 
tMovieList
put tNewLine after tMovieList
set unicodeText of field "movieList" to tMovieList

>> Also, how do I get the proper encoding to use for a string which I get
>> from answer file?  My system is English but the string has Japanese
>> encoded characters and it appears it has to have the language setting
>> as Japanese when encoded?
> To parse shift JIS files, and get as unicode you can use
>
> put uniencode(textoffile,"japanese")

Okay, but how do I determine what to do at runtime?  On my system I 
have three different file names that I am testing with:

1) All english characters
2) All japanese characters
3) Uses european characeters (é)

If I use uniEncode (myfileName, "Japanese") then cases 1 and 2 work but 
not 3.
If I use uniEncode (myFileName, "UTF8") then cases 1 works but not 2 
and 3.
If I use uniEncode (myFileName, "UTF16") then cases 1 and 3 work but 
not 2.

So how do I determine which encoding to use on a string so it displays 
correctly?  Or is there an encoding that I am missing which converts 
correctly every time?  I am still very new to unicode so I apologize if 
I am missing something obvious.

Thanks,

-- 
Trevor DeVore
Blue Mango Multimedia
trevor at mangomultimedia.com