Array Split vs Combine

[Monte Goulding]

It’s probably most likely to do with the string buffer needing to be constantly resized as the array is iterated for combine. Some googling suggests Windows may have issues with this. Our strategy for growing string buffers at the moment is to allocate just enough for the string. Mark would need to chime in on whether growing the buffer exponentially would be suitable. It would result in more memory being allocated than necessary but much less frequent allocations so it depends on what’s most costly as memory gets cheaper.

Cheers

Monte

> On 11 Mar 2021, at 11:34 am, Sean Cole (Pi) via use-livecode <use-livecode at lists.runrev.com> wrote:
> 
> The code for 'Split':
> 
> void MCArraysExecSplit(MCExecContext& ctxt, MCStringRef p_string,
> MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, MCArrayRef&
> r_array)
> {
> if (MCStringSplit(p_string, p_element_delimiter, p_key_delimiter, ctxt .
> GetStringComparisonType(), r_array))
> return;
> 
> ctxt . Throw();
> }
> 
> 
> vs
> 
> The code for 'Combine' (// comments added by me):
> 
> void MCArraysExecCombine(MCExecContext& ctxt, MCArrayRef p_array,
> MCStringRef p_element_delimiter, MCStringRef p_key_delimiter, MCStringRef&
> r_string)
> {
> bool t_success;
> t_success = true;  // Create a register to check progress success
> 
> uindex_t t_count;  // Create a new (t)emp counter for indices
> t_count = MCArrayGetCount(p_array);  // Find out how many arrays there are
> 
> MCAutoStringRef t_string;  // Create (t)emp string to store the result
> if (t_success)
> 
> t_success = MCStringCreateMutable(0, &t_string); // t_success is always
> true here initially and is reset as true if t_string is now mutable, false
> if not
> 
> 
> combine_array_t t_lisctxt;  // create a new array object
> t_lisctxt . elements = nil; // initialise t_lisctxt array
> if (t_success)
> 
> t_success = MCMemoryNewArray(t_count, t_lisctxt . elements); // make sure
> the array was created
> 
> 
> if (t_success)
> {
> 
> t_lisctxt . index = 0;
> 
> MCArrayApply(p_array, list_array_elements, &t_lisctxt);
> 
> qsort(t_lisctxt . elements, t_count, sizeof(array_element_t),
> compare_array_element); // sort the elements
> 
> for(uindex_t i = 0; i < t_count; i++)
> 
> { // Loop through all indices
> 
> MCAutoStringRef t_value_as_string; // create a (t)emp string for element
> value
> 
> 
> 
> t_success = ctxt . ConvertToString(t_lisctxt . elements[i] . value,
> &t_value_as_string); // convert array value to string
> 
> if (!t_success)
> 
> break; skip if unable to convert to string
> 
> 
> t_success =
> 
> (p_key_delimiter == nil ||
> 
> (MCStringAppend(*t_string, MCNameGetString(t_lisctxt . elements[i] . key))
> &&
> 
> MCStringAppend(*t_string, p_key_delimiter)))&&
> 
> MCStringAppend(*t_string, *t_value_as_string) &&
> 
> (i == t_count - 1 ||
> 
> MCStringAppend(*t_string, p_element_delimiter)); // t_success is true if
> the array element and values are added correctly
> 
> 
> if (!t_success)
> 
> break; // skip if unable to add value
> 
> }
> 
> }
> 
> if (t_success)
> 
> t_success = MCStringCopy(*t_string, r_string);  // Copies the (t)emp string
> into the (r)eturn string
> 
> 
> MCMemoryDeleteArray(t_lisctxt . elements);
> 
> if (t_success)
> 
> return;
> 
> 
> // Throw the current error code (since last library call returned false).
> ctxt . Throw();
> }
> 
> 
> Following on from Bob's VM comment, there is reference to
> 'MCMemoryNewArray(t_count,
> t_lisctxt . elements)' which does highlight that some memory management for
> the arrays is necessary in the combine command. This only creates a
> temporary copy of the array for working through. How this plays out
> differently for Windows vs Mac/Linux and why this would be increasing the
> time necessary by a factor of about 4:1 I can't see.
> 
> I've tested as far back as LC7
> (Times - Read into memory, Split to array, Combine from array)
> LC9.5.0 Win64 - 0.437s, 0.516s, 3m 1.378s
> LC9.0.5 Win32 - 0.446s, 0.547s, 3m 27.9s
> LC8.2.0 DP2 - 0.543s, 0.577s, 3m 30.208s
> LC8.0.0 - 0.542s, 0.545s, 3m 30.815s
> LC7.0.0 - 0.827s, 0.460s , 3m 37.896s
> 
> On mac all times are less than 1sec, 3 sec total.
> 
> Sean
> 
> 
> On Wed, 10 Mar 2021 at 17:08, Bob Sneidar via use-livecode <
> use-livecode at lists.runrev.com> wrote:
> 
>> Now THAT is fascinating, considering the Windows performance issues with
>> file access reported in the past. Could it be that combine is somehow
>> caching data to virtual memory?
>> 
>> Bob S
>> 
>> 
>> On Mar 9, 2021, at 1:05 PM, Sean Cole (Pi) via use-livecode <
>> use-livecode at lists.runrev.com<mailto:use-livecode at lists.runrev.com>>
>> wrote:
>> 
>> It's looking to be a Windows only issue. I need to see how far this goes
>> back and then I'll post a bug report. It's making a process that should
>> only take 30s max on a single thread 2GHz remote Win server take 16mins to
>> process 2 of these files, so it will be good to find a solution for this.
>> 
>> Thanks everyone for confirming and providing your input.
>> 
>> Regards
>> Sean
>> 
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode