60 hours divided by 60 is 2 minutes?

[Robert Brenstein]

>Hi all,
>
>I have a 5MB file with about 550000 lines that need to be processed by a
>script. A simple script that deletes a line if the previous line has the
>same contents. That takes more than 60 hours to complete. So I thought I
>divide the file into smaller files of about one 60th of the total number of
>lines. But instead of the expected hour of processing time, it took 2
>minutes for each file to complete.
>
>I understand processes are faster with less data in memory, but I never
>would have thought the difference would be this big.
>
>Any thoughts on how this is possible and what we can learn from it when
>making programs?
>
>Terry
>

My first guess would be that with full file he get into memory 
swapping which is a speed killer. Rev loads all you stack data in 
memory after all. You can test this by setting your virtual memory to 
be just 1 mb over the physical RAM.

Robert