revCopyFile chokes on 1,500 file loop
Sivakatirswami
katir at hindu.org
Fri Aug 4 04:57:39 EDT 2006
platform-network context:
Mac OSX client with Rev app that is "talking to-about" files on an OSX
Server across the LAN. All path stems look like
/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/"
Goal: copy about 1,500 files from one set of directories to another
directory with a new name on the server, not on the client machine,
which is simply initiating, controlling the process remotely.
I'm running this from a small rev card based database with about 800
cards. Each card has the old path to the old file for 1 to 4 files and
some fields from which to build a new file name. The trimmed down script
looks like this, where the Server is mounted on the desktop of the
client machine via AFP.
repeat with x = 1 to the number of cards
go to card x
# do stuff here to build a new file name and set up paths in variables:
repeat for each line y in fld "soundFiles" # each record has a fld
w/list of .wav, .ra, .mp3 files to step through
RevCopyFile (oldpath &"/"& oldFileName), (newpath & "/" & new
file name
# an explicit instance if watching variable watcher would look
something like this:
revCopyFile
"/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/1998/soundclip.ra",\
"/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/1998/ad_1998-08-01_title.ra"
end repeat
# next card....
end repeat
OK, simple enough and appears to work like a charm, and super fast
because no files are being read and written by and from the client
machine, but if we open the directory on the server to which files are
being copied to, we see whole sections of files are missing... it's as
if the script is over-running the ability of the network+server to do
the job.
If I reduce the repeat look to something like
repeat with x = 1 to 10
go to card x
# do stuff here to build a new file name and set up paths in variables:
repeat for each line y in fld "soundFiles" # each record has a list
of .wav, .ra, .mp3 files I step through
RevCopyFile (oldpath &"/"& oldFileName), (newpath & "/" & new
file name
end repeat
end repeat
it works just fine, but if I just let it run from card 1 to 778, the
server can't handle it. Rev rips through the 778 cards about as fast
as it can update the video card to display the next card and, in under 2
minutes it has thrown 1,500 requests at the server to copy and rename
files ranging in size from 300K to 4 Megabytes. Meanwhile we can hear
the mac doing the successful copy beep sound at a frantic pace, so much
so that the speaker can't keep up.
I tried introducing a wait" in the script, but it didn't help. I
assume from the docs that Rev is just doing a shell
cp "old path-filename" "new path-filename" in the background. I'm just
guessing but I think we need to have some kind of feedback in the loop
to make sure the script is not over running some limit on the server
which I don't know what it would be or how to get that feedback data
into the process. If you get the result on each iteration, it is always
empty, so that doesn't provide any progress data. what we need is,
algorithmically stated as if talking to the server:
copy this file to that file
Now I give you some time
You tell me if you are ready?
Can I issue another copy command? I'm listening
OK, yes I heard that, you are done, good
copy next file to next file
Any ideas? Of course I *can* do this in chunks... like maybe set the
repeat for x = 1 to 10 and then wait and do 11-20 etc... and get the
immediate job done, in probably less then 20 minutes, but I'm interested
in solving the problem because I could really use this functionality in
other contexts for some massive global file handling processes.
TIA
Sivakatirswami
More information about the use-livecode
mailing list