Processing Big-ish Data

[FlexibleLearning.com]

Hi Geoff,

Me too :( and yes, 'split/combine' with 'for each' are useful!

Ony processing 10-20Mb here, but parsing into arrays I run out of memory. I
am maxed out for XP (4Gb). Will have to upgrade to Win7 to access more, I
fear, and/or re-write to chunk the data. I didn't realise they were going to
be processing that volume.

With multi-gigs of data, you will certainly have to chunk the data. And
access your progress bar judiciously!

Hugh Senior
FLCO


Geoff Canyon wrote:

I opened my mouth at work when I shouldn't, and now I'm writing a function
to process server log files: multi-gigabytes of data, and tens of million
rows of data. Speed optimization will be key...