Accumulating text is *VERY* slow in LC9 on Windows

Ben Rubinstein benr_mc at cogapp.com
Thu Aug 26 04:49:34 EDT 2021


Hi Alex,

Thanks for responding. It's an interesting idea and I may have to try it.

But I've got 5,000 lines of script that has evolved over 20 years, so I really 
don't want to have to rewrite it all! It seems to me that there is an 
identifiable problem here; something that is happening on (at least one) 
Windows machine, which was fine under LC6 but is broken under LC9; whereas on 
Mac it's not the case.

My hope is that there is some simple solution to fix this!

Ben

On 25/08/2021 19:48, Alex Tweedly via use-livecode wrote:
> Crazy idea - totally untried .... (sorry, I don't have a Win machine)
> 
> 
>    put 1 into tLineCount
>    repeat for each line tRow in tWorkTable
>        put tRow into tNewTable[tLineCount]
>        add 1 to tLineCount
>    end repeat
>    combine tNewTable using CR
> 
> Alex.
> 
> 
> On 25/08/2021 18:15, Ben Rubinstein via use-livecode wrote:
>>
>> Some 20 months ago, I reported that I was in a situation where an app 
>> written in 6.7 needed to be updated to access 64bit drivers, which meant 
>> updating to 9.5 - which displayed horrifying increase in processing time.
>>
>> In fact I was able to put off the evil day - but now it has returned, and 
>> can be put off no longer. A process that normally takes 2 hours is currently 
>> taking 9. The core processing stage has gone from around ten minutes to over 
>> six hours.
>>
>> After way too long, I've finally got down to at least one smoking gun; which 
>> is as simple as can be.
>>
>> Part of what took me so long is a confusion; in production the process runs 
>> on Windows, but I develop on Mac. Although on Mac the overall process does 
>> take about a third longer in LC9 than LC6, the simple tests I've finally 
>> isolated actually run much _quicker_ in LC9 than LC6. So switching between 
>> LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing 
>> signals. But unmistakeably it's *much* slower on Windows.
>>
>> A simple routine which loops over a load of tab and return formatted data 
>> loaded from a TSV file, to truncate a particular field, had the following 
>> results processing a 70MB file of approximately 257,000 rows:
>>
>>     6.7.11 MacOS      9 seconds
>>     6.7.11 Win32     10 seconds
>>     9.6.3  MacOS      2 seconds
>>     9.6.3  Win32    498 seconds
>>
>> I simplified it down to this (pointless) loop which just rebuilds a table 
>> one line at a time:
>>
>>    local tNewTable
>>    repeat for each line tRow in tWorkTable
>>       put tRow & return after tNewTable
>>    end repeat
>>
>> with these results:
>>
>>     6.7.11 MacOS      8 seconds
>>     6.7.11 Win32      7 seconds
>>     9.6.3  MacOS      0 seconds
>>     9.6.3  Win32    591 seconds
>>
>> (there's obviously a lot of variability in these - both were running in IDE, 
>> on a logged-in computer, so stuff was probably going on in the background; 
>> but I know the overall effect is similar when built as standalone and 
>> running by schedule on an unattended machine. But the key thing is: for this 
>> task, LC9 is dramatically slower on Windows!)
>>
>> Have others seen something like this?
>>
>> When I posted about this before (thread: "OMG text processing performance 
>> 6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden 
>> cost of binary<->text transforms. That makes some sense; but given that the 
>> text already exists, I'm wondering whether taking a line out of text would 
>> cause it to be transformed, only to be transformed again when appending? And 
>> in particular, why this would affect Windows only.
>>
>> I have also added tests using "is strictly a binary string" in the code 
>> above, and this was true for neither input 'tWorkTable', nor the output 
>> 'tNewTable', nor any of the 257,00 extracted lines.
>>
>> However it is definitely the accumulating of text that is the issue - simply 
>> looping over the lines - even with testing each one to see if it is 
>> "strictly a binary string" - is a second or less on Windows in LC9.
>>
>> Has anyone had similar experiences? Suggestions for how this could be avoided?
>>
>> Many thanks in advance,
>>
>> Ben
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode




More information about the use-livecode mailing list