Datagrid - GetDataForLine

[Bob Sneidar]

Hi Marty. 

Lotta prior posts on this so I will just summarize. Parsing data out into an array can be a huge pain. It can be cumbersome to parse large amounts of data into arrays. However there are techniques which can make it a great deal faster. 

First, read your data into manageable chunks. If you know your data is on average 500 characters per line, and you want to read roughly 1000 lines at a time, do something like:

read from fileName for 500 * 1000
put it into theData
read from file fileName until return
put it after theData

Now you have a manageable chunk to work with. Next, just repeating for each line may seem the way to go, and some recommend it:

repeat for each line theLine in theData
  put 1 into theColumnCount
  repeat for each item theItem in theLine
    put "column" & theColumnCount into theColumnName
    put theItem into myArray[theRecordCount][theColumnName]
  end repeat
end repeat

Others (myself included) think that simply working with line 1 of theData and then deleting line 1 of theData is the way to go. The reason for this is that each time your refer to a line, say line 256 of theData, the engine has to scan through the block of data until it counts 255 returns, then get the data between that and the next return. Well 256 not so bad. 256,783, bad. By just getting line 1 each time and then deleting it, the engine only has to scan up to the first carriage return each time. 

So if it is a one time parsing, you may as well go ahead and create your array, and then store it as a property and work with that. If you have to parse large amounts of data each and every time you load your app, well there are better ways for sure. 

Also, PLEASE consider learning how to work with databases if you have large data sets. It seems a pain at first, but once you get the hang of it, things will go much better, especially searching and getting subsets of data. 

Bob


On Dec 2, 2011, at 1:04 PM, Marty Knapp wrote:

> I'm using a form datagrid and have it successfully working with a tab delimited list that I cycle through and create an array, then use the FillInData handler in the datagrid to fill it out.
> 
> I am now trying to set this up for large amounts of data. I've read the brief tutorial in the DG manual about using the "dgNumberOfRecords" and "GetDataForLine" but it's geared for accessing data from a database. I did look at the sample stack, but again it's geared for database access.
> 
> The dgNumberOfRecords should be set to the total number of records in my data set, correct?
> 
> In my scenario, how do I use the "GetDataForLine" command?
> 
> Am I wasting time by converting my tab-delimited file to an array?
> 
> I have no database experience and very little knowledge of arrays, so please bear with me! Any sample code, or sample stacks are appreciated.
> 
> Thanks,
> Marty Knapp
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode