Chunks vs Arrays - surprising benchmarking results

[Richard Gaskin]

Paul Looney wrote:
> I have nothing to add directly to the chunk vs array discussion  
> (Trevor's reply was very good) but I have often found it helpful to  
> increase the speed of compound selections by breaking them into  
> individual ones.
> 
> For instance if you have a large database of names and sexes and you  
> want to select every female named "Jan" ("Jan" could be male or female).
> Select all of the Jans first (this will run much faster than the  
> compound selection).
> Then select all of the females from the result of the first selection  
> (this will run faster because it is searching only "Jan"s - a very  
> small list).
> This double selection will run faster than a single compound selection.
> 
> Obviously this requires a known data-set where one filter will  
> eliminate a lot of records (selecting "female", then selecting "Jan"  
> would be much slower in our example because, presumably, half of the  
> list is female and a small portion is Jan).
> On many lists this can create a much bigger speed difference than  
> chunk vs array variance you noted.

One of the tough challenges with this sort of benchmarking is that 
different methods will favor different test cases.

But with delimited rows and columns, I haven't found a way to make a 
two-pass search run faster than one pass, except in very specialized 
cases as you noted.

There's a temptation to use the filter command for the first pass, but 
filter is only faster when testing the first few items; filtering on the 
10th item is much slower, and attempting to test the 50th item in a 
sample data set caused Rev to hang.  RegEx is a harsh mistress.

In my case, I don't often know in advance which item will be searched. 
The queries I'm running usually come from a Search dialog in which the 
user can specify criteria.  I could make the search function smart 
enough to special-case certain types of searches to use a two-pass 
method in which the first pass is the filter command where practical, 
but the overhead of analyzing both the query and the data to make such 
determinations may detract from the benefits of doing so, esp. since my 
continued testing on this is increasingly nudging me toward 
multi-dimensional arrays anyway.  Even with the data bloat and the 
surprising overhead of moving arrays in and out of storage, with a 
little extra work to deal with those the performance of arrays seems 
unbeatable in the broadest range of use cases I've run thus far.

--
  Richard Gaskin
  Fourth World
  Revolution training and consulting: http://www.fourthworld.com
  Webzine for Rev developers: http://www.revjournal.com