getProp syntax query

[Richard Gaskin]

Graham Samuel wrote:

 > On Mon, 2 Apr 2012 Mark Schonewille wrote:
...
 >> You can use only one index, i.e. the setProp/getProp handlers
 >> are one-dimensional.
...
 > Mark, thanks for that. There is absolutely nothing of this in the
 > LC Dictionary AFAICS, although after a search I realised that
 > there is a very hard to read (IMHO) description on section 7.10
 > of the User Guide. I think at the very least a note should be
 > added to the Dictionary entry, so I have submitted one, but I
 > would be very happy if someone such as yourself corrects it.

Or you could do that yourself, with the Comments feature.  Probably a 
useful note to add until that Dict entry gets enhanced.

In fact, it would be great to submit an RQCC request for this Dict 
issue, so it will be on RunRev's radar.


At the heart of this is a core difference between how arrays and custom 
props are handled.  They seem similar but have diverged in recent years:

In the olden days, both LiveCode's associative arrays and the custom 
properties it allows in objects were parallel:  they were one 
dimensional, and you could move data between them easily:

    put the customProperties of tObj into tMyArray
    set the customProperties of tObj to tMyArray

When arrays became multi-dimensional, that made it difficult to use the 
same syntax to accomplish similar goals, so they created a workaround to 
support that which also gave rise to two new functions, arrayEncode and 
arrayDecode.

ArrayEncode translates an array from the memory-specific hash table that 
drives it into a flat form that can be saved to disk.  ArrayDecode 
translates that flattened form back into an array in memory.

But since memory-mapped hash tables can't be used from disk, the format 
of data returned from arrayEncode is not easily traversed, requiring 
considerable clock cycles to translate back into an array when run 
though arrayDecode.

In essence, what those functions do is somewhat similar to what you'd 
have to do if you wanted to flatten array data in a script, walking 
through each element and writing it out into a linear format that 
retains key-value association, nesting, etc.

When you save an array to a custom prop, from what I'm seeing in terms 
of performance it seems that the array data is passed through something 
very similar to the internal routine used for arrayEncode, allowing that 
binary data to be storable, but unfortunately not in a form that's 
easily traversed.

And since those can't be traversed using the same rapid methods that 
memory-based arrays use, we have no ability to use multi-dimensional 
array syntax to address them as properties like we do as arrays.

For the long term I've submitted a request to see if we can come up with 
a b-tree storage method which might allow us to use the same syntax for 
both arrays and nested props:

Make arrays and customProperties behave the same
<http://quality.runrev.com/show_bug.cgi?id=6912>

In the meantime, to access array data stored in a custom prop requires 
two steps, first to extract the array and then to access its nested 
elements:

    put the MyPropArray of tObj into tSomeArray
    put tSomeArray[1][2] into tSomeValue

Note that performance of nested arrays as props is roughly on par with 
running data that had been saved with arrayEncode through the 
arrayEncode function, not nearly as fast as using a true array in a 
variable.  So for performance-critical uses it may be helpful to extract 
the array early on and use it from a global, storing it back only when 
needed.


This is a good opportunity to share a bit of geeky fun news with you folks:

Kevin Miller and Ben Beaumont have generously asked Mark Waddingham to 
take a moment away from his very busy schedule to provide a description 
of the format of the data we get from arrayEncode.

I had requested this because I have a few cases where I need to be able 
to extract a specific data element from an array stored on disk, without 
the overhead of reading the entire file and running it through 
arrayDecode.   This is for a CGI where every millisecond counts, and 
some of the arrays I'm working with have hundreds of thousands of nested 
keys.

To justify the time required for Mark to do this, we made a deal:  Mark 
would provide a sparse description only sufficient to get me started so 
he could get back to more critical tasks, provided I take that info and 
flesh it out into an article for LiveCode Journal which explains the 
format in detail and provides a working example of how to use it.

Mark came through on his end of the bargain wonderfully, providing a 
sample function that worked on the first level of array keys, which was 
enough for me to expand it to be able extract the data from any key 
regardless of depth.

The article explaining it all is in progress now and close to done.  I 
expect to be able to post it to LiveCode Journal by Saturday.

Here's a tantalizing tidbit to whet your appetite:

The sample stack I wrote for this creates a three-dimensional array with 
100 items in each dimension, resulting in an arrayEncoded file with 1 
million elements that's about 20 MBs on disk.

The function I made from Mark's example code can extract the data from 
any element in that file in about 3 to 5 milliseconds.

Hint:  the function makes heavy use of the binaryDecode function and the 
seek command, both of which are explained in depth in the article and 
which you may find very valuable for a wide variety of uses.  The seek 
command is a god for rapidly traversing large files easily.

I look forward to sharing this with you ASAP -

--
  Richard Gaskin
  Fourth World
  LiveCode training and consulting: http://www.fourthworld.com
  Webzine for LiveCode developers: http://www.LiveCodeJournal.com
  LiveCode Journal blog: http://LiveCodejournal.com/blog.irv