"read...until <string>" -- buffer size

[Richard Gaskin]

The "read...until <string>" form is wonderfully convenient, but very slow.

It's so slow, in fact, that I've found I can write a few dozen lines of 
code to perform a functionally identical task at much greater speed.

The algo I use I picked up from some old HyperCard article back in the 
day.  In short, I read from disk into a buffer by a specified amount, 
then walk through the lines within the buffer in memory.  When I reach 
the last item, which doesn't have a terminator, I read another batch 
from the file, appending my buffer, and repeat this process until I've 
completed my traversal of the file.

While this has given me a satisfying speed bump well worth the time it 
took to write it, it occurs to me that it shouldn't be necessary at all.

After all, it's not like "read...until <string>" is reading only one 
byte at a time from disk; I haven't read that part of the source but I'd 
be surprised if it's not reading at least one block's worth in each pass 
(usually 4k, depending on the file system).

So in essence, it would seem the current "read...until <string>" algo in 
the engine is nearly identical to what I'm doing in script, with only 
one critical difference:  the buffer size.

Oddly enough, experimenting with different buffer sizes has yielded 
surprising results.  At first I thought that minimizing disk access 
would be the primary boost, so I tried reading in 1 MB chunks but found 
far greater performance with much smaller amounts, even though it meant 
touching the disk more often.  Ultimately, it seems the optimal buffer 
size in my experiments on one system was 128k; when reading smaller 
amounts the extra disk accesses take a toll, and when reading larger 
amounts it's also slower, perhaps due to malloc anomalies or the way LC 
uses malloc in that context.

All this leaves me with a proposal:

Should we consider adding an optional argument for "read...until 
<string>" to specify the buffer size the engine will use?

E.g.:

   read from file tMyFile until CR with buffer 128000

...or:

   read from file tMyFile until CR for 128000

...or if we're going to try to be more English-like about it:

   read from file tMyFile until CR for 128k


Worth submitting as an enhancement request?

Anyone here in a position to implement this themselves?

And is there anyone here who happens to know the buffer size the engine 
currently uses for this?

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com