Filtering array vs plain list

[Dick Kriesel]

On 2/1/08 2:36 PM, "Shari" <shari at gypsyware.com> wrote:

> I'm trying to determine the best way to store a humungous block of
> data for both speed and accessibility.
> 
> My original plan was to do an array something like this:
> 
> Item #2 is an identifier unique to each individual.  It will not be a
> number, but text + a number.
> 
> census[Atlanta,5489,name] = John
> census[Atlanta,5489,religion] = Baptist
> census[Atlanta,5489,birthYear] = 1976
> census[Atlanta,5489,baseballTeam] = Braves
> census[Atlanta,5489,eyeColor] = blue
> census[Atlanta,9988,name] = Judy
> census[Atlanta,9988,religion] = Mormon
> census[Atlanta,9988,birthYear] = 1926
> census[Atlanta,9988,baseballTeam] = Yankees
> census[Atlanta,9988,eyeColor] = green
> census[Chicago,3258,name] = Billy
> census[Chicago,3258,religion] = Atheist
> census[Chicago,3258,birthYear] = 1982
> census[Chicago,3258,eyeColor] = blue
> census[Chicago,3258,baseballTeam] = Red Sox
> 
> Alternately of course this could be a list with each line dedicated
> to the individual.
> 
> Each individual would have 300-400 different items and there will be
> thousands of individuals.
> 
> The data needs to be accessed very very quickly, which points to an
> array as above.  And most of the time very specific pieces of data
> will be accessed and updated.  And changes will be made constantly.
> So far, so good.
> 
> But here's the glitch.
> 
> I need to also be able to make global changes to the list, for
> example, update everybody whose baseball team is "Red Sox" and
> birthYear is 1950.  For example, change all instances of Red Sox to
> White Sox if the birthYear is 1950.
> 
> Is there any way short of looking inside each element of thousands of
> keys for the matching birthYear/baseballTeam?
> 
> This would presumably be very slow.  But if the array were a list
> with thousands and thousands of lines, each line having 300-400
> items, wouldn't that be slow to access as well?
> 
> What am I missing?
> 
> Shari

Hi, Shari.  There are other ways to structure your data that can make your
"glitch" easier to handle.  Choosing the best way depends on various
factors, such as:
1) time constraints on adding, updating, and deleting people, properties,
and values
2) whether you have multiple simultaneous users
3) your time and skills

But jumping the gun, you could consider using arrays like the following:

birthYear[1976] = 5489
baseballTeam["Red Sox"] = 3258
eyeColor["blue"] = 5489 & cr & 3258

Then evaluating your compound example involves retrieving the values of
baseballTeam["Red Sox"] and birthYear[1950], deriving the intersection of
those values, removing them from baseballTeam["Red Sox"], and inserting them
into baseballTeam["White Sox"].

I've used this approach; it's fast.  I'd be glad to help you further if
you're interested.

-- Dick