revCopyFile chokes on 1,500 file loop

Sivakatirswami katir at hindu.org
Fri Aug 4 04:57:39 EDT 2006


platform-network context: 

Mac OSX client with  Rev app that is "talking to-about" files on an OSX 
Server across the LAN.  All  path stems look like

/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/"

Goal: copy about 1,500 files from one set of directories to another 
directory with a new name on the server, not on the client machine,  
which is simply initiating, controlling the process remotely.

I'm running this from a small rev card based database with about 800 
cards. Each card has the old path to the old file for 1 to 4 files and 
some fields from which to build a new file name. The trimmed down script 
looks like this, where the Server  is mounted  on the desktop of the 
client machine via AFP.

repeat with x = 1 to the number of  cards

go to card x
   # do stuff here to build a new file name and set up paths in variables:

    repeat for each line y in fld "soundFiles" # each record has a fld 
w/list of  .wav, .ra, .mp3 files to step through
       RevCopyFile (oldpath &"/"& oldFileName), (newpath & "/" & new 
file name

# an explicit instance if watching variable watcher  would  look 
something like this:
revCopyFile 
"/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/1998/soundclip.ra",\
"/Volumes/Server/WWW/*Sites/www.himalayanacademy.com/html/audio/1998/ad_1998-08-01_title.ra"

   end repeat
# next card....

end repeat

OK, simple enough and appears to work like a charm, and super fast 
because no  files are being read and written by and from the client 
machine,  but if we open the directory on the server to which files are 
being copied to, we see whole sections of files are missing... it's as 
if the script is  over-running the ability of the network+server to  do 
the job.

If I reduce the repeat look to something like

repeat with x = 1 to 10

go to card x
   # do stuff here to build a new file name and set up paths in variables:

    repeat for each line y in fld "soundFiles" # each record has a list 
of  .wav, .ra, .mp3 files I step through
       RevCopyFile (oldpath &"/"& oldFileName), (newpath & "/" & new 
file name
   end repeat

end repeat

it works just fine, but if I just let it run from card 1 to 778, the 
server can't handle it. Rev  rips  through the 778 cards about as fast 
as it can update the video card to display the next card and, in under 2 
minutes it has thrown 1,500 requests at the server to  copy and rename 
files ranging in size from 300K to 4 Megabytes. Meanwhile we can hear 
the mac doing the successful copy beep sound at a frantic pace, so much 
so that the  speaker can't keep up.

I tried introducing a  wait" in the script,  but it  didn't help. I 
assume from the docs that  Rev is just  doing a shell

cp "old path-filename" "new path-filename"  in the background.  I'm just 
guessing but I think we need to have some kind  of feedback in the loop 
to make  sure the script is  not over running some limit on the server 
which I don't know what  it  would be or how to get that feedback data 
into the process. If you get the result on each iteration, it is always 
empty, so that doesn't provide any progress data. what we need is, 
algorithmically stated as if talking to the server:

copy this file  to that file
Now  I give you some time
You tell me if  you are ready?
Can I issue another copy command? I'm  listening
OK, yes I heard that, you are done, good
copy next file to next file


Any ideas? Of course I *can* do this  in chunks... like maybe set the 
repeat for  x = 1 to 10 and then wait and do 11-20 etc... and get the 
immediate job done, in probably less then 20 minutes, but I'm interested 
in solving the problem because I could really use this  functionality in 
other contexts for some massive global file handling processes.

TIA

Sivakatirswami













More information about the use-livecode mailing list