UPDATE: filterArray()

[Bob Sneidar]

I’m surprised about the time differential. Obviously LC parsing is much faster, I guess because filterArray has the initial overhead of creating and populating the database. I’ve never used it for large datasets. 

I guess the main advantage for using filterArray() is that using the LC parser, for each use case would require it’s own code. filterArray() is a one-stop-shop approach. For instance, if you need to compare data with another dataset (as in an SQL join) or finding the maximum value in a dataset, or limiting your results. 

Also, with queries you can do complex sort criteria and complex joins, return calculations as columns, etc. All of these use cases would require a different code set for the LC parser. Yes with SQL the extra code would be writing the SQL statement, but for anyone proficient with SQL, that’s easy enough. 

It’s conceivable that you could write a pseudo-database system that uses LC arrays instead of sql, but you would have to do auto saves to file on that kind of database to prevent data loss, and that would . 

LIKE queries are also one of the slowest kinds of queries there are, so any query that does not use an index is going to take a longer time. I suppose I could add indexing but that would create more first time overhead. 

I made the assertion when I first wrote this, that for single queries, the sql method would not be very performant, because the overhead of creating the database to begin with could be significant, but if subsequent queries were made directly on the database, the filterArray method would begin to catch up, because the LC parser method cannot gain efficiency with iteration. I suppose the real test would be to see whether or not direct queries agains a memory database is faster than the LC parser. 

I do have a findInArray() function as well, but it’s fairly simple, returning only the keys that match a value in a single column. 

Bob S


> On May 17, 2024, at 6:55 PM, Richard Gaskin via use-livecode <use-livecode at lists.runrev.com> wrote:
> 
> Thank you, Bob.
> 
> It reminded me of a test I started a while back, and prompted me to finish the relevant part of it:
> 
> go url "https://fourthworld.net/lc/Array%20vs%20SQLite.livecode"
> 
> 
> Richard Gaskin
> FourthWorld.com
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode