OMG text processing performance 6.7 - 9.5
neville.smythe at optusnet.com.au
Tue Feb 4 21:40:33 EST 2020
Just for interest, and to see just how slow lineOffset is, I added a couple of more tests to the search for occurrences of “Valjean” in the Gutenberg English translation of Les Miserables. I also wanted find how filter performs.
The searches were first applied to the raw binary text as read from the utf-8 encoded file, without using textDecode; then on the text utf-8 decoded
Parse 0 : using itemdelimiter ‘Valjean’ (case insensitive)
Parse 1: using offset with skips
Parse 2: using offset, truncating the text and 0 skip
Parse 3: use lineOffset with skips
Parse 4: use lineOffset, truncating the text and 0 skip
filter: use filter to find lines containing '*Valjean*'
Parse 1 and 2 produced 1120 hits. Parse 0 gave 1121 hits, the extra one being a false positive at the end of the file, which needs to be accounted for in an implementation. I was slightly surprised that the character offsets produced were the same for raw and for utf-8 text, I guess I was expecting the latter to give the unicode character offset. Parse 3 and 4 and filter all output 1099 lines.
searches on raw text:
parse0 11 ms
parse1 9 ms
parse2 751 ms
parse3 2551 ms
parse4 753 ms
filter 16 ms
searches on utf-8 text:
parse0 4386 ms
parse1 224367 ms
parse2 3461 ms
parse3 636554 ms —— !!!!
parse4 7242 ms
filter 2258 ms
So for long texts it is best to use raw binary text and search with character offset(pSt,pSrc,skip) [Parse 1]. If you have to search on utf-8 encode text then use Parse 2, deleting initial sections of the text as you go. Never use lineOffset (except for small text) even if that means extra code to find line endings on either side of the character offset when you really want the found line. If you don’t actually need the offset of the hits in the original file - for example for editing the original - then filter is the fastest on long text and just as fast on short text, but depending on your needs you probably have to do another search on the filtered text; but this would be a viable approach if the number of lines produced is itself small.
More information about the use-livecode