Performance Problem with large file

[Len Morgan]

Paul Looney wrote:

> > If it is taking 3 seconds now, processing from a variable should take 
> > (far) less than one second. If that is so, then you will not need to 
> > worry about covering the "hole".
>   
If 3 seconds was the case when the field is displayed, I would not have 
even brought it up.  It's 3 seconds if the field is made invisible.  
It's over a minute (about 3 seconds PER RECORD) if the field is showing.
> > I do second Richard's suggestion about moving from XML; it is really 
> > slow, takes more memory for code, takes more memory for data, etc. - 
> > worth asking the vendor if it is necessary.
>   

> Some vendors won't listen. WebMerge works with a lot of merchant data 
> feeds, and some of the vendors are so fixated on the fashionability of 
> XML that even the argument about the impact in their service usage 
> doesn't sway them.

In my case, a) the vendor WON'T change and b) I only have to generate an XML file.  I don't have to read or process it.  To me, I just take a csv text file and then add all the XML bits and spit it out.  Other than the declarations at the top and bottom of the file, every record simply takes three items off each line and wraps XML tags around them before outputing them.  Really pretty simple.

> > Also, are you using "repeat with" where you could be using "repeat 
> > for"? The latter is about ten times faster.
>   

> Paul raises a very important point here; "repeat for each" is a 
> performance godsend.

> One great thing about this list is that if you'll post the code you'll 
>have a small army of seasoned pros rewrite it for you for free.  :) 

> Len, can you post that handler?

Ask and ye shall receive.  Here's the "Convert" mouseUp script:

    #####################################################################################

    # Clear the output field we're going to use and turn if off (for performance reasons)
    put empty into field "xmlOutput"  
    put xmlFileHeader() after field "xmlOutput"
    set the visible of field "xmlOutput" to false
    
    REPEAT for each line curLine in field "txtInput"
        put the item 1 of curLine into idnum
        put item 2 of curLine into lst_n
        put item 3 of curLine into fst_n
        put item 4 of curLine into day_in
        get xmlWriteNewInmateRecord(idnum, lst_n, fst_n, day_in, tCount)
        get setPercentDone(tCount, ttlLines)
        put "Record " & tCount & " of " & ttlLines into field "progressText"
        put tCount + 1 into tCount
    END REPEAT
     
    # Write the closing tag
    put "</RCATS>" & cr after field "xmlOutput"
    set the visible of field "xmlOutput" to true

#############################################################################################

That's the "meat" of the handler.  Hopefully most of it will be self-explanitory.  The only part that might not be obvious is the setPercentDone function which is given a record number and total records and update a progress bar.  Hope that sheds some light.

Thanks again,

Len