Making "read from file" less blocky.

[Dar Scott Consulting]

OK. Skip "cat" for now.  Everything else should work on Windows.


To load a file in one swoop...
put URL("binfile:" & ThisFile) into IntoThisVariable

And if you saved the file with compress, then decompress it.


To load portions of a file just-in-time...
Use "at" and "for" in "read from"

That wont help if you immediately want to find the average or something. But it might help if you want to read a header and first record, then move to another record as needed.


I am awful at explaining things so twist my arm until I do it right.


> On Aug 4, 2019, at 5:28 PM, Tom Glod via use-livecode <use-livecode at lists.runrev.com> wrote:
> 
> heheh.... its always a fun exercise.
> 
> Dar, these are linux based solutions right? using windows here at the
> moment, so I can't test, but when I test and optimize my application for
> linux i will try these.
> 
> On Sun, Aug 4, 2019 at 4:43 PM dsc--- via use-livecode <
> use-livecode at lists.runrev.com> wrote:
> 
>> Even more "really out there" of the "really out there".
>> 
>> Don't read in the file.
>> 
>> Access portions JIT, that is, lazily.
>> 
>> Create a function that pulls in segments of the file. Kinda like this:
>> 
>>        function segmentOfFile pStartIndex, pEndIndex, pThisFile
>> 
>> Or this:
>> 
>>        function segmentOfCurrentFile pStartIndex, pEndIndex
>> 
>> Dar
>> 
>>> On Aug 4, 2019, at 1:22 PM, dsc--- via use-livecode <
>> use-livecode at lists.runrev.com> wrote:
>>> 
>>> More "really out there".
>>> 
>>> I like the idea of trying to speed up an upfront foreground load.
>> Something simple like this:
>>> 
>>> put blah-plah into IntoThisVariable.
>>> 
>>> where blah-blah is nana-nana or decompress( nana-nana )
>>> where nana-nana is one of these:
>>>      URL ("binfile:" & ThisFile)
>>>      shell("cat " & ThisFile)
>>> 
>>> Function decompress() makes two RAM hits and requires control over the
>> loaded files.
>>> 
>>> 
>>>> On Aug 4, 2019, at 12:47 PM, Dar Scott Consulting via use-livecode <
>> use-livecode at lists.runrev.com> wrote:
>>>> 
>>>> I love "really out there".  I wanna play.
>>>> 
>>>> At the start of any solution, try this. It might speed up any method
>> but would take some time at the start.
>>>> 
>>>> get shell( "cat " & ThisFile & " > dev/null" )
>>>> 
>>>> I think that is likely to pre-load the system file buffers for you.
>>>> 
>>>> If one is feeling adventurous, one can try open process (cat) to avoid
>> the wait; it will probably move through the file faster than the script and
>> sectors will already be loaded when you ask for them.
>>>> 
>>>> Dar
>>>> 
>>>>> On Aug 4, 2019, at 7:59 AM, Alex Tweedly via use-livecode <
>> use-livecode at lists.runrev.com> wrote:
>>>>> 
>>>>> OK, here's a "really out there" suggestion ....
>>>>> 
>>>>> 1. Run a local web server  to serve files (locally only).
>>>>> 
>>>>>   Can be done various ways, including (easily) via LC and the httpd
>> library,
>>>>> 
>>>>>      (build that server as a standalone and have it running - started
>> from your app if need be...)
>>>>> 
>>>>> 2. in your stack, just do
>>>>> 
>>>>> load url ("http://localhost:8080/myfilename") with message
>> "mycallback"
>>>>> 
>>>>> and handle the file once it has been read in the "mycallback" handler
>>>>> 
>>>>> -- Alex.
>>>>> 
>>>>> 
>>>>> On 04/08/2019 01:56, Tom Glod via use-livecode wrote:
>>>>>> Hey folks,
>>>>>> 
>>>>>> I'm having trouble finding a combination of settings that allows my
>> file
>>>>>> loading  to seem to happen in the background.
>>>>>> 
>>>>>> repeat while read_result is not "eof"
>>>>>>       read from file ThisFile for (1024 * 1000) bytes
>>>>>>       put the result into read_result
>>>>>>       put it after IntoThisVariable
>>>>>>       add length(it) to amount_read
>>>>>>       TSTProgress amount_read,ExpectedSize,"%","Loading File ..."
>>>>>> (Progress Indicator Handler)
>>>>>>       wait 10 milliseconds with messages
>>>>>>    end repeat
>>>>>> 
>>>>>> no matter what I try, its still sluggish, and it seems like messages
>> are
>>>>>> still accumilating instead of being processed by the engine.
>>>>>> 
>>>>>> Am I missing something?  Normally waiting with messages sufficiently
>> frees
>>>>>> the engine to allow the UI to remain responsive.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Tom
>>>>>> _______________________________________________
>>>>>> use-livecode mailing list
>>>>>> use-livecode at lists.runrev.com
>>>>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>>>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>>> 
>>>>> _______________________________________________
>>>>> use-livecode mailing list
>>>>> use-livecode at lists.runrev.com
>>>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> use-livecode mailing list
>>>> use-livecode at lists.runrev.com
>>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> use-livecode mailing list
>>> use-livecode at lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> 
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode