60 hours divided by 60 is 2 minutes?

[Rob Cozens]

>Any thoughts on how this is possible and what we can learn from it when
>making programs?

Hi Terry,

Before dealing specifically with 1 @ 60 hours vs 60 @ 2 minutes, I 
need to know if you are actually deleting each duplicate line on the 
fly?  Have you tried building a new list instead:

functon purgeDuplicates @textData
    put empty into newData
    put numToChar(30) into lastLine -- any char not in the first line 
of textData
    repeat for each line thisLine in textData
       if thisLine = lastLine then next repeat
       put thisLine&return after newData
       put thisLine into lastLine
    end repeat
    return newData -- or write as a command and "put newData into textData"
end purgeDuplicates

I'd be curious to know what algorithm you used and what times the 
above handler produces.

Other things to look at:

1.  Is it possible you are maxed out in actual RAM and spending a lot 
of time reading from/writing to virtual memory?

2.  Are you passing the 55K lines of text by value or reference?

3.  Have you tried writing your handler inline with the handler that 
reads in the data so it needn't be passed at all?

Eg:
     put get URL (whatever) into textData
     put empty into newData
     put numToChar(30) into lastLine -- any char not in the first line
     repeat for each line thisLine in textData
       if thisLine = lastLine then next repeat
       put thisLine&return after newData
       put thisLine into lastLine
    end repeat
    put newData into textData

instead of

     put get URL (whatever) into textData
     put purgeDuplicates(textData) into textData

(although if textData is passed by reference, the impact of item 
three is negligible).
-- 

Rob Cozens
CCW, Serendipity Software Company
http://www.oenolog.com/who.htm

"And I, which was two fooles, do so grow three;
Who are a little wise, the best fooles bee."

from "The Triple Foole" by John Donne (1572-1631)