60 hours divided by 60 is 2 minutes?

[Jan Schenkel]

--- MultiCopy Rotterdam-Zuid <info at multicopy.org>
wrote:
> Hi all,
> 
> I have a 5MB file with about 550000 lines that need
> to be processed by a
> script. A simple script that deletes a line if the
> previous line has the
> same contents. That takes more than 60 hours to
> complete. So I thought I
> divide the file into smaller files of about one 60th
> of the total number of
> lines. But instead of the expected hour of
> processing time, it took 2
> minutes for each file to complete.
> 
> I understand processes are faster with less data in
> memory, but I never
> would have thought the difference would be this big.
> 
> Any thoughts on how this is possible and what we can
> learn from it when
> making programs?
> 
> Terry
> 

Hi Terry,

Though in extreme cases it might have to do with the
OS swapping the memory to disk at an incredible rate,
I'm more inclined to believe that it might have
something to do with the algorithm.
Off the top of my head, I'd process it with:

function ReadUniqueLinesFromFile pFile
  put URL pFile into tInput
  put empty into tPrevLine
  repeat for each line tLine of tInput
    if tLine <> tPrevLine then
      put tLine & return after tOutput
      put tLine into tPrevLine
    end if
  end repeat
  delete char -1 of tOutput
  return tOutput
end ReadUniqueLinesFromFile

And that should work pretty quickly.

Hope this helped,

Jan Schenkel.

=====
"As we grow older, we grow both wiser and more foolish at the same time."  (La Rochefoucauld)

__________________________________________________
Do you Yahoo!?
Y! Web Hosting - Let the expert host your web site
http://webhosting.yahoo.com/