"read...until <string>" -- buffer size

[Richard Gaskin]

Kay C Lan wrote:

 > ...I'd suggest the solution is not to manually set the buffer size
 > but to have LC find the optimum for you.
 >
 > With the current variety of disk sizes, proportion of disk empty, rpm
 > and even if it's mechanical or SSD, what ever you discover to be the
 > optimum on your machine is unlikely to be the optimum on Richmond's
 > heroic G3 tray loading iMac running 10.3. I'd even suggest taht
 > whatever you find is the optimum today, may not be the optimum in a
 > year. I've bench tested many HDs and there seems to be quite a variety
 > of factors that determine the file size that achieves the fastest
 > read/write times, and quite a variety in those file sizes, so I'm sure
 > whether it's buffer size, something else, or a combinateion, the same
 > would be true for 'read from file'.
 >
 > Wouldn't it be better if you could, in an openStack or similar:
 >
 > get maxReadSpeed()
 > --runs a suite of tests to determine the settings that achieves
 > maximum read from file speed.
 > --then sets these as defaults which will be used by read (and other?) 
functions
 > --only needs to run once
 > --automatically reset to pre-LC settings when you Quit LC
 >
 > And from then on use the standard read functions.

I like the fresh thinking on this, but if it's difficult for me to find 
an optimal buffer size for a given hardware configuration and file 
system, will be it any easier for the engine team to address not only 
those concerns but also my own application-specific needs?

I'm not all that picky about syntax, and both options require the engine 
team to expose some underlying mechanism that affects buffer size.  As 
long as it gets the job done I'm not all that particular about how it 
gets done.

With "put...after..." Dr. Raney was able to change the way he was using 
pointers with malloc to deliver a many-fold speed boost with no extra 
work required by us at all.  If that were possible here I certainly 
wouldn't mind, and maybe simply increasing the existing buffer size 
might do the trick*.

Looking at what's been proposed thus far, it seems like:

Good: The default buffer size is increased.

Better:  The default buffer size is increased, along with some means
     of allowing the scripter to optionally adjust it manually if
     they choose for the needs of a particular application.

Best: RunRev figures out a way to optimize for my application's needs
     for me, and provides some means of allowing us to make that happen.


 > So a bug report to fix the current speed, then if it's found that
 > speed improvements can be had by adjusting system/LC internal
 > settings, an enhancement request for an LC function that would
 > determine and set these as default. This would ensure whether you're
 > running a nearly full 250GB 5400rpm HD or a basically empty 750GB SSD,
 > ' read from file tMyFile until CR' will work faster than anything you
 > can mangle using multiple lines of LC code.

Indeed, I don't think it's unreasonable to expect that now, at least as 
far as a modest increase to the buffer size that better reflects modern 
file systems and RAM configurations.

But like many other cases (e.g. regex vs filter vs repeat for each, or 
arrays vs simple lists) what's optimal may be specific to a given 
use-case, so I'm not yet certain which is the "best" approach.


As a side note:

Anyone else here ever notice that this is valid syntax?:

    read from file tMyFile for 2 uint4

With the read command, other valid binary chunk types include uint1, 
uint2, and their plurals, uint1s, uint2s, uint4s (filter the Dictionary 
by "uint" to find these, or see the notes in the Parameters section for 
the Read command).

I bring these up because they represent one of the great strengths of 
LiveCode over so many other xTalks:

LiveCode contains many elements that go far beyond the relatively narrow 
bounds of HyperTalk, with most of its users blissfully enjoying the 
tokens they use without ever having to think about the rest.

But if you find yourself doing deeper work on binary files, or needing 
to parse files larger than can fit in RAM, or a wide range of other 
tasks that programmers sometimes have to do, it's nice to discover these 
optional gems that make such coding as enjoyable as "go next".

Like arrays, regex, and many other language elements that have become 
popular over the years since they were introduced to xTalk in LiveCode, 
all of these help LiveCode address a far larger scope of viable projects 
than its predecessors.

Sometimes a little branching out from one's roots is a good, healthy 
evolution.  Kevin's recent video on Open Language and Widgets takes that 
further, and while it may seem a bit daunting to think about memory 
management at that level, for those who need it it's every bit as 
liberating as being able to navigate between cards or create a button.



* Once again I tried going through GitHub to find a specific relevant 
bit of code, and once again it seems I lack the training needed to do so 
effectively.  Anyone here know where the current "read...until" buffer 
size is defined?  Knowing the current buffer size would be very helpful 
for making suggestions about changing it.

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com