How to filter a big list

Brian Yennie briany at qldlearning.com
Mon Oct 19 19:28:01 EDT 2009


I would suspect that the leading "*" is causing the most slowdown.  
That is going to force the filter command to search the entire line  
every time, since the "['-]" portion could be anywhere.

Switching to SQLLite could work, but you are going to have to  
completely reformat your data. If you just throw your lines into  
records in a database, the problem will be the same.

Maybe there is a higher level solution, for example:

1) Move your searchable strings to the beginning of each line (filter  
on myString* will be much faster)

2) Sort your data and process it in chunks, showing the results as  
they come in (total time will be the same, but better user experience)

3) Try using lineOffset() to find one match at a time?

HTH

> Thank you Sarah, Chris and Andre,
>
> I use already a variable and my filter pattern is "*[' -]" &  
> myString & "*".
>
> I'm going to try with a SQLite database.
>
>
> Le 19 oct. 2009 à 23:49, Sarah Reichelt a écrit :
>
>> On Tue, Oct 20, 2009 at 6:49 AM, Jérôme Rosat <jrosat at mac.com> wrote:
>>> I wish to filter a list which contains approximately 300'000  
>>> lines. I try
>>> the "filter ... with" command. It's slow.
>>>
>>> I try the "Repeat for each" loop but it's slower.
>>>
>>> Is it possible to use the "filter ... with" command and to "force"  
>>> RunRev to
>>> check only one "item" of the line and not the whole line with the
>>> "filterPattern" ?
>>
>>
>> What filter pattern are you using? I have found filter to be very  
>> fast
>> even with large lists, provided the filter pattern is simple.
>> e.g. filter with "*blue*" is fast but filter with "*" & tab & "*" &
>> tab & "*blue* will be slow.
>>
>> If this is the sort of thing you are trying, then I suggest a
>> preliminary fast filter to get the overall length down, then maybe
>> changing to a "repeat for each" loop to refine the filter.
>> And as Chris said, make sure you are always operating on a variable,
>> not the contents of a field.
>>
>> Cheers,



More information about the use-livecode mailing list