Sorting text is *VERY* slow in LC9 on Windows (Re: Accumulating text is *VERY* slow in LC9 on Windows)

[Ben Rubinstein]

Hi Mark,

Thank you for this, very promising.

Only two things puzzled me. One you've already addressed when you corrected 
the specified cheese.

The other was
> The only caveat is that it might cause apps mutating lots of medium->large 
> strings concurrently to take up more memory in general... > ... (and any issue arising from that could be resolved by moving to the 64-bit
> windows engine).

I have been doing my tests on the 64-bit Windows engine. What am I missing?

TIA (and T as always for all your work),

Ben

On 10/09/2021 14:06, Mark Waddingham via use-livecode wrote:
> On 2021-09-02 18:38, Mark Waddingham via use-livecode wrote:
>> We will endeavour to fix for 9.6.5-rc-1 (due 'real soon now'!).
> 
> So I have been prodding the windows 'accumulating large strings' speed problem 
> this week (in amongst other things).
> 
> It is definitely memory allocation causing the problem.
> 
> The rule for extending a string buffer when inserting/appending more text is 
> pretty much the same as 6.7 (the amount of 'extra space' it asks for each time 
> is a little less - but changing it to the constant used in 6.7 makes no 
> difference). What is different between 6.7 and 7.0+ is the memory allocation 
> patterns of the engine itself and I think it is this which is invariably 
> causing a whole new buffer having to be allocated when appending to large 
> strings, rather than the existing one being extended.
> 
> The windows heap is much more prudent than UNIXy counterparts it would seem - 
> where UNIX heaps will happily leave plenty of free space (which the heaps know 
> about and thus can re-use), Windows appears to avoid that like the plague 
> (which I'm sure is the case for lots of historical reasons and backwards 
> compatibility). [ To give a very rough analogy, the map of used space in a 
> heap on windows is like a block of cheddar; whereas on UNIXy systems it will 
> be like a block of edam ].
> 
> So anyway, long story short, making a string buffer grow in proportion to its 
> size appears to mitigate the problem - I still need to finesse things a bit 
> though to ensure that memory starvation doesn't occur when you have a couple 
> of large mutating strings but all being well we should be able to get 
> something together for 9.6.5-rc-1.
> 
> The only caveat is that it might cause apps mutating lots of medium->large 
> strings concurrently to take up more memory in general (i.e. in that extra 
> unused 'just in case' space at the end of each buffer) but no more than the 
> same apps would on any other (UNIX-based) platform (and any issue arising from 
> that could be resolved by moving to the 64-bit windows engine).
> 
> For reference, the bug about the string accumulation problem as posed by Ben 
> is https://quality.livecode.com/show_bug.cgi?id=23319 (this will also fix the 
> sort speed too - as the final step the sort command performs internally is 
> re-accumulating the string in the new order).
> 
> Warmest Regards,
> 
> Mark.
> 
> P.S. I cannot really say whether this will help with the various 'IDE can be 
> like treacle' on Windows problems - but it definitely can't hurt!
>