Okay, I think I'm brain damaged -- nope, I figured it out

[Geoff Canyon]

On Thu, Sep 12, 2013 at 6:06 PM, Mark Wieder <mwieder at ahsoftware.net> wrote:

> OTOH, this scales linearly and is faster than "before", "after", or
> "into": on a half-fast dual-core linux box I get 103 ticks for a
> million iterations. Even with the overhead of the handler call, it
> looks like this avoids making a copy of the original.
>


This does avoid making a copy, but so does put after.

tl;dr: use put after, and don't use variables to work with more than about
100mb unless you have no choice.

I tested each of the options discussed in this thread. Each test iterates
with an ever-longer target string, until an iteration takes more than a
second to complete.

Here are the timings I got. For each:
 -- the first column is the iteration
 -- the second is the size of the resulting variable at that step
 -- the third is the elapsed time
 -- the fourth is the ratio of the time to the previous iteration
 -- the fifth is the ratio of the size of the resulting string to the time
taken for that iteration.

None of these test scale against the size of the append string (it's the
same across all the tests and iterations) -- rather, they all test scale
against the size of the target string, which grows at a constant rate for
each of  the tests. For that, constant time would be ideal. For example,
within the boundaries of LC's math functions, addition is (nearly) constant
against the size of the target, meaning that on my machine, this takes
about a second regardless of which line I uncomment:

on mouseUp
   --put 1 into X
   --put 999999 into X
   put the long seconds into T
   repeat 9999999
      add 1 to X
   end repeat
   put the long seconds - T
end mouseUp


Appending to a string doesn't work that way. The longer the target string
is, the longer the append to it takes. The good news is that it's nowhere
near linear, except for the first two options.



before
         put "xxxxxxxxxx" before R

putting a string before a variable is slow, likely because it is either
duplicating the whole string, or because it is shoving over the entire
string to make room at the front. This only got up to a string 800k long
before taking more than a second to iterate. It appears to be roughly
linear on the size of the target string.

1    100000    0.077    0.00006    1298703
2    200000    0.207173    2.690563    965376
3    300000    0.387425    1.870054    774344
4    400000    0.522814    1.349459    765090
5    500000    0.665046    1.272051    751828
6    600000    0.810649    1.218937    740148
7    700000    0.953222    1.175875    734351
8    800000    1.111549    1.166097    719716



put into
         put R & "xxxxxxxxxx" into R

putting a string before a variable is also slow, because it likely is
duplicating the whole string. This also only got up to a string 800k long
before taking more than a second to iterate. It appears to be roughly
linear on the size of the target string.



1    100001    0.07383    0.011054    1354479
2    200001    0.189246    2.563273    1056830
3    300001    0.367313    1.940926    816745
4    400001    0.500568    1.362783    799094
5    500001    0.645846    1.290227    774180
6    600001    0.787196    1.21886    762200
7    700001    0.917061    1.164972    763309
8    800001    1.053472    1.148748    759395



append
         append "xxxxxxxxxx", R

command append pNew, @pString
    put pNew after pString
end append

An append function is much faster. Note that in the previous two tests,
each iteration only appended 100k. This test appended 5mb -- 50x as much --
in each iteration, and also gets up to a 45mb target string before hitting
one second per iteration. The @ means the target is not being duplicated,
but calling the function is still overhead that can be avoided.

1    5000000    0.572176    0.01905    8738570
2    10000000    0.631929    1.104431    15824565
3    15000000    0.684285    1.082851    21920686
4    20000000    0.742656    1.085302    26930369
5    25000000    0.816506    1.09944    30618272
6    30000000    0.869896    1.065388    34486884
7    35000000    0.92884    1.06776    37681403
8    40000000    0.981937    1.057165    40735814
9    45000000    1.022531    1.041341    44008444



put after
         put "xxxxxxxxxx" after R

The reigning champ. This is also appending 5mb each iteration, and the
target string hits 85mb before stopping. Still, this is showing substantial
slowing: the last 5mb takes about 17 times as long to append as the second
5mb.

1    5000000    0.114283    0.006386    43751098
2    10000000    0.174828    1.529784    57199058
3    15000000    0.243715    1.394027    61547287
4    20000000    0.29158    1.196397    68591819
5    25000000    0.334841    1.148368    74662300
6    30000000    0.402346    1.201603    74562709
7    35000000    0.469453    1.16679    74554839
8    40000000    0.507524    1.081096    78814005
9    45000000    0.576554    1.136013    78049923
10    50000000    0.625147    1.084281    79981206
11    55000000    0.6839    1.093983    80421100
12    60000000    0.738292    1.079532    81268660
13    65000000    0.812725    1.100818    79977846
14    70000000    0.849118    1.044779    82438484
15    75000000    0.879976    1.036341    85229617
16    80000000    0.931517    1.058571    85881406
17    85000000    1.004347    1.078184    84632097