externals

[Richard Gaskin]

Randall wrote:

 > Richard wrote:
 >> I've posted the example to RevNet - in Rev, choose
 >> Development->Plugins->GoRevNet, and in RevNet look for
 >> "4W Folder Watcher" in the Stacks section.
 >>
 >> This modest example lets you pick a folder and it checks
 >> for changes to files (name, size, additions, deletions).
 >> It also provides a slider to govern the interval between
 >> checks so you can experiment to find the best balance
 >> between system overhead and responsiveness.
 >
 > Richard, how big is the folder you are monitoring?  How nested?

Not nested at all, as I noted in my post.  As currently written, it 
checks only for file changes in the target folder.

 > I have written similar code of course.

Of course.

 > But it doesnt scale.

True, but just about everything in computing involves tradeoffs.  The 
method that scales least efficiently at runtime may be the most 
efficient one to develop, and the one which scales best may be 
cost-prohibitive to develop.  The key is to find the most appropriate 
balance based on the specific needs of a given application, and the 
budget driving it.

For the low cost of just a few minutes' time, polling in Rev will give 
acceptable performance in some cases, and perhaps unacceptable 
performance in others:

On a folder with about 5000 files, it takes more than 400ms on a 2.16GHz 
Mac. But with the 250 files in my Preferences folder it takes only about 
four milliseconds, and with about a hundred files it takes less than one 
millisecond.

I have many folders with fewer than a hundred files, and very few with 
more than a thousand; whether it's worth the orders-of-magnitude greater 
cost to develop a solution that can more gracefully account for 
unusually large sets of files can only be determined by the specifics 
governing the project at hand, the age-old tug of war between scope and 
budget.


 > And by definition it isnt real time.

Define "real time".

By definition all event messages will be sent after the fact since they 
describe something that just happened, and many are derived from polling 
(devices, processes, etc.) which makes them even slower.  The only 
question is how long after the fact will be acceptable for a given use.

It's of course more efficient to poll as deep as possible, and in the 
case of file changes it may be that digging into the kernal or even the 
storage device driver with Assembler will yield the best performance. 
But that sort of work can get quite expensive, so it seemed at least 
worthwhile to find out how responsive this can be in a very-high-level 
4GL like Rev.

Many file changes are the result of a user action, either dragging a 
file into a folder, saving a file from within an application, etc. 
Given that people are much slower than computers, anything that happens 
in response to a user event will have a lot of leeway in terms of 
perceived responsiveness.

True, that doesn't account for all possible cases, such as applications 
or processes that generate files on their own.  But it may be useful for 
some folks anyway, so I posted it.

The nice thing about doing this in Rev is that the engine is pretty fast 
with string parsing, such as the result one gets back from "the detailed 
files".  The bigger latency is obtaining that list from the OS, which 
any polling method will be subject to (and hence I appreciate your 
desire to move beyond polling, but which means beyond Rev).

In short, polling "the detailed files" may prove suitable for a good 
many uses, if not yours perhaps for others here.



Forgive me if you've already described this, but what exactly does your 
application need to do that requires the shortest possible notification 
time for file changes?

And given that task, what would be the threshold of acceptable 
performance to satisfy your project's goals?  Would the OS APIs suffice, 
or will you need to go deeper?



Innovation is rarely trivial to accomplish. Visionaries must usually 
bind their vision with sweat to realize their goals.

Jan presented a reasonable decision tree for your project's success:

     As for having any suggestions? How about these:
     - learn C and even some x86 assembler
     - learn more about file systems
     - learn more about operating systems
     - research how to tap into the I/O parts of the kernel
     OR
     - try to find and hire someone with the above skills
       to implement this for you
     - have your checkbook ready and figure in a hefty
       maintenance cost of the code when planning the project

Keep us posted when you release.  Sounds like an interesting project.

--
  Richard Gaskin
  Fourth World
  Revolution training and consulting: http://www.fourthworld.com
  Webzine for Rev developers: http://www.revjournal.com