Processing Big-ish Data

Phil Davis revdev at pdslabs.net
Thu Sep 12 10:58:59 EDT 2013


Big data = big fun!  :-)

I enjoy those challenges because it (when successful) showcases man's 
ability to bring the machine into his own service, instead of the other 
way around. At least that's what I tell myself.

Phil Davis


On 9/12/13 6:46 AM, Geoff Canyon wrote:
> On Thu, Sep 12, 2013 at 2:35 AM, FlexibleLearning.com <
> admin at flexiblelearning.com> wrote:
>
>> With multi-gigs of data, you will certainly have to chunk the data. And
>> access your progress bar judiciously!
>>
>
> Indeed. One of my co-workers was munging/aggregating using SQL queries --
> not a good idea, and was still not done after an hour. Out of curiosity I
> got him to send me his data before lunch, and my first pass yesterday
> worked by processing all the (test) data into an array, and finished in
> about 90 seconds. But that was just 100mb of data. Then I found out how big
> the data set really is, and started re-writing. Right now I have something
> that isn't chunked yet, but can be, and is much faster, but doesn't quite
> work ;-)
>
> Debugging in my future...
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>

-- 
Phil Davis





More information about the use-livecode mailing list