best data format

[Kee Nethery]

On Mar 28, 2011, at 7:39 PM, Todd Geist wrote:

> thanks guys.
> 
> Now a couple of follow up questions.
> 
> Where is a good place to stick all this data?  custom Props on substack?

It depends upon how quickly it changes.

If it changes infrequently, I'll pull it and store it in a field along with a timestamp for the entire data set. When the timestamp is older than some period of time, I pull it again. Otherwise the field where I store the data is my data cache. If I quit the app and restart, it is still there.

If it is a bunch of rows where some change, and there are way more rows in the source table than I'll ever need in a day, I store the data with a time to live value. I keep the data in a variable. Before I pull from the data, I sort by time to live time and delete oldest stuff until my dataset is within a set number of characters in size (don't want to use all the RAM for any one data set). Then I delete any row with an old time to live value (which is going to be at one end of the data set because of the time to live sort). Then I search for the desired row. If it is not there, I grab it from the database and add it to the data set. Then I pull the data from the dataset. What normally happens is that the most frequently used stuff stays cached and the less frequently used stuff gets added to the cache and disappears eventually.

Then, there is data that gets changed frequently or data where I must have the absolute most recent data and for that, I pull it from the database and throw it away when I'm done.

If it's tons of data that I am creating and only I am using until it is completely processed, I have a getter and a setter that store and pull the data from disk in a file. Once I throw it back onto disk, I set that variable to empty to clear out the RAM. 

I personally like using fields for slow changing data. Lets me see what the data is. If I don;t want others to see, I can always make the field invisible for the version going out to others.

Kee