shift-JIS and Unicode

[Ron]

Hi Tuviah
> 
>> It may end up being a lot of work to use old stacks
>> containing Japanese, since it looks like we've got to recode the scrips
>> as well, but I haven't really played with them much yet.
> Well if we gave you guys what you wanted, which is to deal with
> shift-JIS, multibyte, it wouldn't be cross platform, and you would be
> unsure if a character was indeed one or two bytes. We're also in the
> process of modifying the uniencode and unidecode property so you will be
> able to easily convert form shift-JIS, utf8 to unicode and vice versa.

I think that you are going in the right direction here. It will be very
helpful to have this ability within RR itself. I agree that we should not
expect you to _use_ SJIS/JIS but it will be great if we can do something
like:
answer file ""
put "binfile:"&it into tUrl
put URL tUrl into thetext
put unidecode(thetext,SJIS) into convertedtext
set the unicodeText of field 1 to convertedtext
 
> When typing plain text (ASCII character 0 to 255) the engine actually
> saves that as one byte per character. If you want
> to know for sure, it's always good to check the textfont of a selection.
> Even when setting the unicodetext, the engine optimizes. It also always
> saves
> *carriage* returns, tabs,  as one byte per character. This is to save
> memory, hard disk space, and speed up processing. Rather than bloat
> everything, and go with everything is a unicode character like Java, we
> decided to find a way for unicode to fit in comfortably with the ASCII

a wise decision that helps us in the long run.

> 
> For those who may be wishing to write some unicode processing algorithms,
> they can check out the useunicode property, and numtochar and chartonum
> routines. I would be interested in hearing what if any special challenges
> people have run into when modifying their programs to deal with unicode.

I'll let you know as soon as I give it a thorough workout. Thanks again
Ron