Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)

[Mark Waddingham]

On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote:
> We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!).

So I have been prodding the windows 'accumulating large strings' speed 
problem this week (in amongst other things).

It is definitely memory allocation causing the problem.

The rule for extending a string buffer when inserting/appending more 
text is pretty much the same as 6.7 (the amount of 'extra space' it asks 
for each time is a little less - but changing it to the constant used in 
6.7 makes no difference). What is different between 6.7 and 7.0+ is the 
memory allocation patterns of the engine itself and I think it is this 
which is invariably causing a whole new buffer having to be allocated 
when appending to large strings, rather than the existing one being 
extended.

The windows heap is much more prudent than UNIXy counterparts it would 
seem - where UNIX heaps will happily leave plenty of free space (which 
the heaps know about and thus can re-use), Windows appears to avoid that 
like the plague (which I'm sure is the case for lots of historical 
reasons and backwards compatibility). [ To give a very rough analogy, 
the map of used space in a heap on windows is like a block of cheddar; 
whereas on UNIXy systems it will be like a block of edam ].

So anyway, long story short, making a string buffer grow in proportion 
to its size appears to mitigate the problem - I still need to finesse 
things a bit though to ensure that memory starvation doesn't occur when 
you have a couple of large mutating strings but all being well we should 
be able to get something together for 9.6.5-rc-1.

The only caveat is that it might cause apps mutating lots of 
medium->large strings concurrently to take up more memory in general 
(i.e. in that extra unused 'just in case' space at the end of each 
buffer) but no more than the same apps would on any other (UNIX-based) 
platform (and any issue arising from that could be resolved by moving to 
the 64-bit windows engine).

For reference, the bug about the string accumulation problem as posed by 
Ben is https://quality.livecode.com/show_bug.cgi?id=23319 (this will 
also fix the sort speed too - as the final step the sort command 
performs internally is re-accumulating the string in the new order).

Warmest Regards,

Mark.

P.S. I cannot really say whether this will help with the various 'IDE 
can be like treacle' on Windows problems - but it definitely can't hurt!

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps