valueDiff for arrays?

[Mark Waddingham]

On 2018-08-04 21:00, Richard Gaskin via use-livecode wrote:
> Mark Waddingham wrote:
> 
>> Yes - so come up with LCS handlers which do that, share them, let's
>> work together to make them as efficient in LCS as possible and then
>> see what we can do in the engine to improve their performance (which
>> I'm guessing is as much the purpose for the proposition as the
>> syntax).
> 
> It is, primarily.  The minor convenience is nice too, but you know how
> much I love to see performance enhancements in the engine.

:)

The important thing you have to remember is that you only get a 
performance advantage for hand-coding code like that in the engine *if* 
the overhead of dispatching script dominates the processing being done. 
That can actually be really hard to determine from inspection.

Especially if things require hash-key lookups (i.e. name creation) - 
then you might find you get a speed up for things involving small 
numbers of keys - but virtually none for large numbers of keys 
(particularly as the key size goes up).

So you then have to balance whether its 'worth it' from a C++ code 
maintenance point of view.

Certainly, I guess a metric could be 'how much is a particularly utility 
function used'. i.e. If a utility function gets into a common library, 
and we find that it is used 'all the time' across many different 
projects *then* it should probably be put on the list for being 
implemented in the engine (and deserve English-like syntax in the 
process!).

> 
> As for implementation, Mark Wieder's looks good:
> http://lists.runrev.com/pipermail/use-livecode/2018-August/248862.html
> 
> Thinking about performance, I wonder if there's anything from some of
> the changes that have boosted PHP 7's performance so far above its
> earlier versions which may be relevant for LC:
> https://www.reddit.com/r/PHP/comments/3q2brz/how_is_php_7_twice_as_fast/

I'd actually be really interested in a direct speed comparison between 
exactly equivalent operations in PHP7 and LC.

The reason is that (from my previous relatively terse, admittedly, 
review of PHP7's implementation details) LC actually does mostly the 
same things - and has done since 7. All of it related to the 
introduction of libfoundation (which I actually started designing, and 
started implemented about 7/8 years ago - long before it appeared in a 
release - although the seeds for it were planted long before - 
namification of handler names, and literals in the engine at version 5 
or 6 being a key precursor).

There is actually (and always has been) quite a long list of things 
which I've always wanted to do with libfoundation to improve its 
performance vastly - pleasingly many of those ideas are more than 
validated by PHP7 (as it actually does them!). However, most of them 
have a huge ripple effect on a substantial portion of the code - and 
that's a significant issue.

For example - MCNameRef which is a 'uniqued string' (IS_STR_INTERNED in 
PHP7 speak) used for all literals, handler names, array keys etc. is a 
separate value type to MCString. This was a design error (born of the 
previous rendering of MCNameRefs pre-7) with hindsight - it should have 
been a kind of MCString.

Similarly, integer array keys should be integers - and not rendered as 
strings at all unless needed.

These two things are relatively easy to do at the libfoundation level, 
but they require a significant API change - which I don't think can be 
made backwards compatible which means updating all uses. None the 
updating work is necessarily hard - but it needs to be done carefully 
and needs to be exceptionally well planned so it can be done rapidly and 
all in one go as it would be really difficult to maintain two separate 
branches whilst it was being done (honestly 7 was a complete nightmare 
from that point of view - I don't even want to think about the amount of 
man hours that went into just keeping the 6 and develop-7 branches in 
sync!!).

Other things (like storing short strings in the value memory block, 
rather than in a separate array) I think could be done without any 
changes to the libfoundation API.

I also experimented with (small) integers being 'tagged pointers' at one 
point - which would eliminate the overhead of most integer operations. 
However, as it turned out the idea on its own didn't really make a great 
deal of difference - what did make a difference was a value pool 
allocator - number values 'live fast and die young' which means that if 
you cache the most recently freed number value, and re-use that you 
eliminate their overhead. That same pattern worked exceptionally well 
for other values too (short strings, for example).

Of course, tagged integers would probably make a substantial difference 
if array keys didn't have to be names - the performance of even 
hash-table based sequences would be far faster.

Anyway, I could probably go on at length about this but will stop there.

As a point of comparison, the reason why PHP7 has these things and LC 
does not (yet) is simply because I think you'll find that PHP7 has (1) 
taken a great deal longer to get any of it than LC did (LC started 
getting these things at 7)... And PHP has quite a number more engineers 
involved in it at the C++ level than I think we do...

Warmest Regards,

Mark.

P.S. I do need to dig into PHP7's exact details more - it is something 
I've only had brief amounts of time to do so - at the very least it 
should indicate what does and does not work and also help see how we 
could do them better :)

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps