lines of UFT16 text are broken?

[Kee Nethery]

On Mar 30, 2011, at 5:58 AM, Trevor DeVore wrote:

> On Tue, Mar 29, 2011 at 9:04 PM, Kee Nethery <kee at kagi.com> wrote:
> 
>> How do people deal with this? Do I need to build a UTF16 version of all the
>> text parsing routines to safely get each line?
>> 
> 
> Can you iterate over the lines of the UTF8 text and then convert to UTF16
> when you are done?

The problem I am running into is that the data is actually a set of items that are delimited by their position on a line.

123456789012345678901234567890
c1data       c2data           c3data

where (for example) c1data is from character 1 to 6, c2data is from 8 to 17, c3data is from 19 to 30

The problem is that (in my example) the don't symbol is a single character and they write it into a specific character position. But ... when it gets saved out as UTF8 it takes up 3 characters. I figured that if I converted it to UTF16 I'd at least have a good chance of counting correctly by knowing each character was 2 bytes but alas, that is not how livecode works (at least for me).

I know, the people who create positional data formats should be <insert bodily harm description here> because of this and other issues, but it's what I have and they aren't changing it any time soon far as I can see.

Thanks for the suggestion,
Kee