60 hours divided by 60 is 2 minutes?

[Richard Gaskin]

David Vaughan wrote:

>> I have a 5MB file with about 550000 lines that need to be processed by
>> a
>> script. A simple script that deletes a line if the previous line has
>> the
>> same contents. That takes more than 60 hours to complete. So I thought
>> I
>> divide the file into smaller files of about one 60th of the total
>> number of
>> lines. But instead of the expected hour of processing time, it took 2
>> minutes for each file to complete.
> 
> Terry
> 
> I am a bit puzzled by your result in the first place. I generated
> 550000 lines with random data which had some chance of duplication in
> the next line. I then processed it to remove duplicates. The latter
> task took a whole four seconds. Not two minutes and not 60 hours; for
> the whole file, not for one sixtieth. Were you using "repeat for each"?

Also, when it comes to adding or deleting, working with arrays is much
faster than with large chunks.  Remember that you can use arays and chunks
interhangeably with the split and combine commands.
 
-- 
 Richard Gaskin 
 Fourth World Media Corporation
 Custom Software and Web Development for All Major Platforms
 Developer of WebMerge 2.0: Publish any database on any site
 ___________________________________________________________
 Ambassador at FourthWorld.com       http://www.FourthWorld.com
 Tel: 323-225-3717                       AIM: FourthWorldInc