Unicode from Variable?

[Devin Asay]

On Nov 25, 2008, at 3:16 PM, Scott Rossi wrote:

> Thanks for the response Devin, but I'm still unclear how you DO  
> anything
> with the data stored in the variable item offsets, line offsets,
> comparisons, etc).  A kind listmember clued me into the fact that  
> you must
> first uniEncode the thing you want to find in the variable, and I  
> have since
> found that by uniDecoding the variable I can get at my English/Roman
> content.

I'll confess, that most of my work in unicode has involved displaying  
rather than analyzing or manipulating the text in any complex way. I  
worked on a way to sort Cyrillic unicode, but it was a killer, and I  
never got it exactly right. I think in general you're going to have a  
hard time doing some of this stuff in variables, since there is no  
unicodeText property of variables, only the raw strings of bytes. I  
tend to put everything in a field, even if it's hidden, and do stuff  
that way. Statements like

line 2 of the unicodeText of fld "foo"
word 3 to 5 of the unicodeText of fld "bar"

seem to work just fine. Given that, if I were searching for strings in  
unicode text, I'd probably do it something like:

put the unicodeText of fld "searchterm" into tSrchStr
put lineOffset(tSrchStr,the unicodeText of fld "searchcontainer") into  
tLineNum

You could also try doing everything with UTF-8:

put unidecode(the unicodeText of fld "searchTerm","UTF8" into tSrchStr
put unidecode(the unicodeText of fld "searchcontainer","UTF8") into  
tSrchCont
put lineOffset(tSrchStr,tSrchCont) into tLineNum

This is all untested. Just my hunch given what I've learned about how  
unicode works in Rev. If you need to, say, select said found unicode  
text in a field, it could get dicey.

>  But it seems that doing all the standard variable manipulation
> we're used to is much more complex/fragile when dealing with Unicode.

That's pretty much the takeaway message. Everything is doable, but  
more fragile and complex. That said, once you figure it out, it's  
pretty stable and reliable.
>
>
> In any event, thanks for the pointers to the stacks.  I found several
> example stacks around the 'net (including your site), and though  
> each seems
> to have some examples that don't work properly, the majority of info  
> is
> helpful.

If you found any of my examples that didn't work, I'd love to know, if  
you get a chance to pass them on.

HTH

Devin


Devin Asay
Humanities Technology and Research Support Center
Brigham Young University