OMG text processing performance 6.7 - 9.5

[Neville]

Just for interest, and to see just how slow lineOffset is, I added a couple of more tests to the search for occurrences of “Valjean” in the Gutenberg English translation of Les Miserables. I also wanted find how filter performs.

The searches were first applied to the raw binary text as read from the utf-8 encoded file, without using textDecode; then on the text utf-8 decoded

Parse 0 : using itemdelimiter  ‘Valjean’ (case insensitive)
Parse 1: using offset with skips
Parse 2: using offset, truncating the text and 0 skip
Parse 3: use lineOffset with skips
Parse 4: use lineOffset, truncating the text and 0 skip
filter: use filter to find lines containing '*Valjean*'

Parse 1 and 2 produced 1120 hits. Parse 0 gave 1121 hits, the extra one being a false positive at the end of the file, which needs to be accounted for in an implementation. I was slightly surprised that the character offsets produced were the same for raw and for utf-8 text, I guess I was expecting the latter to give the unicode character offset. Parse 3 and 4 and filter all output 1099 lines. 

Results:

searches on raw text:

parse0      11 ms
parse1        9 ms
parse2    751 ms
parse3  2551 ms
parse4    753 ms
filter          16 ms

searches on utf-8 text:

parse0        4386 ms
parse1    224367 ms
parse2        3461 ms
parse3    636554 ms —— !!!!
parse4        7242 ms
filter            2258 ms

So for long texts it is best to use raw binary text and search with character offset(pSt,pSrc,skip) [Parse 1]. If you have to search on utf-8 encode text then use Parse 2, deleting initial sections of the text as you go. Never use lineOffset (except for small text) even if that means extra code to find line endings on either side of the character offset when you really want the found line. If you don’t actually need the offset of the hits in the original file - for example for editing the original - then filter is the fastest on long text and just as fast on short text, but depending on your needs you probably have to do another search on the filtered text; but this would be a viable approach if the number of lines produced is itself small.

Neville