Processing Big-ish Data

Geoff Canyon gcanyon at gmail.com
Thu Sep 12 09:46:34 EDT 2013


On Thu, Sep 12, 2013 at 2:35 AM, FlexibleLearning.com <
admin at flexiblelearning.com> wrote:

> With multi-gigs of data, you will certainly have to chunk the data. And
> access your progress bar judiciously!
>


Indeed. One of my co-workers was munging/aggregating using SQL queries --
not a good idea, and was still not done after an hour. Out of curiosity I
got him to send me his data before lunch, and my first pass yesterday
worked by processing all the (test) data into an array, and finished in
about 90 seconds. But that was just 100mb of data. Then I found out how big
the data set really is, and started re-writing. Right now I have something
that isn't chunked yet, but can be, and is much faster, but doesn't quite
work ;-)

Debugging in my future...



More information about the use-livecode mailing list