Processing Big-ish Data

[Lynn Fredricks]

 > Indeed. One of my co-workers was munging/aggregating using 
> SQL queries -- not a good idea, and was still not done after 
> an hour. Out of curiosity I got him to send me his data 
> before lunch, and my first pass yesterday worked by 
> processing all the (test) data into an array, and finished in 
> about 90 seconds. But that was just 100mb of data. Then I 
> found out how big the data set really is, and started 
> re-writing. Right now I have something that isn't chunked 
> yet, but can be, and is much faster, but doesn't quite work ;-)

I don't know if Ruslan is going to chime in or not, but we've had to deal
with that with Valentina DB. I don't know if there's a solution that would
work for your situation, but it is something our customers, especially the
financial services industry ones, have dealt with.

Best regards,

Lynn Fredricks
President
Paradigma Software
http://www.paradigmasoft.com

Valentina SQL Server: The Ultra-fast, Royalty Free Database Server