Need Help Throttling Downloads From an FTP Site

[Scott Rossi]

How large are the files you're retrieving?  If the script below is your
actual script, you might try allowing some execution time in the loop:

repeat with each line remoteFilePath in listOfFilePaths
    -- set new localFileName is set before the download request is made
    put url ("ftp://anonymous:myEmailAddress@ftp.sec.gov/" & remoteFilePath)
into url ("file:/" & exportFolderPath & "/" & localFileName )
    wait 2 seconds with messages --  <-- ADD THIS
end repeat

It would probably be most helpful to you to check the status of each
request, so you can keep track of which events succeeded and which failed. I
imagine there are folks on the list who have something like this more
readily available than me.

Regards,

Scott Rossi
Creative Director
Tactile Media, UX/UI Design



On 9/21/15, 2:33 PM, "use-livecode on behalf of Gregory Lypny"
<use-livecode-bounces at lists.runrev.com on behalf of
gregory.lypny at videotron.ca> wrote:

> Hello everyone,
> 
> I posted about this a while back but am still having trouble.
> 
> I need to download thousands of files from the Security and Exchange
> Commission's website. Access is through anonymous FTP with "anonymous" as the
> username and my email address as the password. I've been using Put in a Repeat
> With loop as
> 
> repeat with each line remoteFilePath in listOfFilePaths
> ‹ set new localFileName is set before the download request is made
>     put url ("ftp://anonymous:myEmailAddress@ftp.sec.gov/" & remoteFilePath)
> into url ("file:/" & exportFolderPath & "/" & localFileName )
> end repeat
> 
> but my script dies (the stack is lifeless and unresponsive) after a few dozen,
> and sometimes a few hundred downloads. I used similar scripts in Mathematica
> and confirmed that the problem is session-timed-out and
> cannot-connect-to-server types of errors. The SEC's webmaster tells me, "There
> is no load/rate limiting on FTP, but if you are running a fast process, it is
> possible you are temporarily overwhelming the server." So, I'm thinking that I
> need to throttle my requests, and maybe should be using libURLDownloadToFile
> to check the status of the current file being downloaded and not request
> another file until the current download is complete. I also wonder whether I
> should be connecting to the FTP site only once with the username and password,
> loop my requests, and then close the connection. Not sure how to do either of
> these and would greatly appreciate any suggestions or tips.
> 
> Gregory
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode