OMG text processing performance 6.7 - 9.5

[Ben Rubinstein]

Ooh, that's very likely to be a good tip.

(I think the database stuff is a red herring, by the way: the bulk of the 
delay is in the processing stage, which is just reading text files in and out.)

Thanks everyone for their input. I'll report back with what I can find (unless 
I can persuade IT to let me use the 32 bit drivers!)

Ben

On 30/01/2020 22:41, Ralph DiMola via use-livecode wrote:
> I found this as well. Another thing, it's faster to truncate the string and
> search from the beginning than using a "start at" on the entire string when
> searching for all occurrences of a string . This was counter intuitive to me
> until Mark explained that skipping chars requires more work because
> repetitive skipping of Unicode chars is slower than many "memcpy"s on very
> long strings.
> 
> Ralph DiMola
> IT Director
> Evergreen Information Services
> rdimola at evergreeninfo.net
> 
> 
> -----Original Message-----
> From: use-livecode [mailto:use-livecode-bounces at lists.runrev.com] On Behalf
> Of Neville via use-livecode
> Sent: Thursday, January 30, 2020 4:49 PM
> To: use-livecode at lists.runrev.com
> Cc: Neville
> Subject: Re: OMG text processing performance 6.7 - 9.5
> 
> Are you perchance using lineOffset searches? I have found that lineOffset
> performance on utf8 text degrades exponentially with the length of the file,
> presumably as it searches for line breaks. Use offset instead which remains
> fast (and much faster still if you can search on the raw text before
> textencoding, then utf8 encode the found chunks)
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>