There should be a "unique" option on sort . . .

[Richard Gaskin]

Peter Haworth wrote:

> Hi Richard,
> It's not so much the arithmetic operation or putting a value into the array
> element, it's the repeat loop itself which I don't think is needed.  It can
> be replaced with "combine pData with return and return".  That one line
> gets rid of the duplicates.  So the function would look like this:
>
> function UniqueListFromArray2 pData
>    local tKeys
>    combine pData with return and return
>    put the keys of pdata into tKeys
>    sort tKeys
>    return tKeys
> end UniqueListFromArray2
>
> I came up with that because the OP mentioned that an engine-embedded way to
> de-dup would be faster than any user written handler, which seems like a
> reasonable assumption.
>
> Pretty academic really because it seems like all the solutions work
> acceptably fast but nevertheless interesting to compare.

Excellent!

I added a new function to the mix which uses the split command, but at 
first I found that it was failing the sanity check - the results were 
different.

After looking over the results I discovered an interesting anomaly:  the 
split command is inherently case sensitive, and turning the caseSenstive 
to false within a handler that uses it has no effect.

Is that a bug?

So the only way to use split is when you want to a case-sensitive 
results, and if so you'll need to set the caseSensitive to true for both 
functions since it governs the sort results:

function UniqueListFromArray3 pData
    set the caseSensitive to true
    split pData using cr and cr
    put the keys of pData into tKeys
    sort lines of tKeys
    return tKeys
end UniqueListFromArray3

So when case sensitivity will do what you need, the result is pretty 
impressive - about half the time required for the other fastest option:

10 iterations on 10000 lines of 5 or fewer chars:
"Add 1" Array: 58 ms (5.8 ms per iteration)
"Split" Array: 24 ms (2.4 ms per iteration)
Results match - Each list has 997 lines

--
  Richard Gaskin
  Fourth World
  LiveCode training and consulting: http://www.fourthworld.com
  Webzine for LiveCode developers: http://www.LiveCodeJournal.com
  Follow me on Twitter:  http://twitter.com/FourthWorldSys