Accumulating text is *VERY* slow in LC9 on Windows
Ben Rubinstein
benr_mc at cogapp.com
Thu Aug 26 04:49:34 EDT 2021
Hi Alex,
Thanks for responding. It's an interesting idea and I may have to try it.
But I've got 5,000 lines of script that has evolved over 20 years, so I really
don't want to have to rewrite it all! It seems to me that there is an
identifiable problem here; something that is happening on (at least one)
Windows machine, which was fine under LC6 but is broken under LC9; whereas on
Mac it's not the case.
My hope is that there is some simple solution to fix this!
Ben
On 25/08/2021 19:48, Alex Tweedly via use-livecode wrote:
> Crazy idea - totally untried .... (sorry, I don't have a Win machine)
>
>
> put 1 into tLineCount
> repeat for each line tRow in tWorkTable
> put tRow into tNewTable[tLineCount]
> add 1 to tLineCount
> end repeat
> combine tNewTable using CR
>
> Alex.
>
>
> On 25/08/2021 18:15, Ben Rubinstein via use-livecode wrote:
>>
>> Some 20 months ago, I reported that I was in a situation where an app
>> written in 6.7 needed to be updated to access 64bit drivers, which meant
>> updating to 9.5 - which displayed horrifying increase in processing time.
>>
>> In fact I was able to put off the evil day - but now it has returned, and
>> can be put off no longer. A process that normally takes 2 hours is currently
>> taking 9. The core processing stage has gone from around ten minutes to over
>> six hours.
>>
>> After way too long, I've finally got down to at least one smoking gun; which
>> is as simple as can be.
>>
>> Part of what took me so long is a confusion; in production the process runs
>> on Windows, but I develop on Mac. Although on Mac the overall process does
>> take about a third longer in LC9 than LC6, the simple tests I've finally
>> isolated actually run much _quicker_ in LC9 than LC6. So switching between
>> LC6 and LC9 on Mac as I tried to isolate the issue was giving confusing
>> signals. But unmistakeably it's *much* slower on Windows.
>>
>> A simple routine which loops over a load of tab and return formatted data
>> loaded from a TSV file, to truncate a particular field, had the following
>> results processing a 70MB file of approximately 257,000 rows:
>>
>> 6.7.11 MacOS 9 seconds
>> 6.7.11 Win32 10 seconds
>> 9.6.3 MacOS 2 seconds
>> 9.6.3 Win32 498 seconds
>>
>> I simplified it down to this (pointless) loop which just rebuilds a table
>> one line at a time:
>>
>> local tNewTable
>> repeat for each line tRow in tWorkTable
>> put tRow & return after tNewTable
>> end repeat
>>
>> with these results:
>>
>> 6.7.11 MacOS 8 seconds
>> 6.7.11 Win32 7 seconds
>> 9.6.3 MacOS 0 seconds
>> 9.6.3 Win32 591 seconds
>>
>> (there's obviously a lot of variability in these - both were running in IDE,
>> on a logged-in computer, so stuff was probably going on in the background;
>> but I know the overall effect is similar when built as standalone and
>> running by schedule on an unattended machine. But the key thing is: for this
>> task, LC9 is dramatically slower on Windows!)
>>
>> Have others seen something like this?
>>
>> When I posted about this before (thread: "OMG text processing performance
>> 6.7 - 9.5") Mark Waddingham suggested that it might be to do with a hidden
>> cost of binary<->text transforms. That makes some sense; but given that the
>> text already exists, I'm wondering whether taking a line out of text would
>> cause it to be transformed, only to be transformed again when appending? And
>> in particular, why this would affect Windows only.
>>
>> I have also added tests using "is strictly a binary string" in the code
>> above, and this was true for neither input 'tWorkTable', nor the output
>> 'tNewTable', nor any of the 257,00 extracted lines.
>>
>> However it is definitely the accumulating of text that is the issue - simply
>> looping over the lines - even with testing each one to see if it is
>> "strictly a binary string" - is a second or less on Windows in LC9.
>>
>> Has anyone had similar experiences? Suggestions for how this could be avoided?
>>
>> Many thanks in advance,
>>
>> Ben
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
More information about the use-livecode
mailing list