Keys of a custom property array

[Richard Gaskin]

Monte Goulding wrote:

 > I agree it makes total sense to use the same format when
 > reading/writing to disk. As all the data is loaded into memory anyway
 > I wonder how much extra time it would take to turn it into a pointer
 > that was just as fast to access as a regular array.... perhaps that
 > could be done at first access?

ArrayEncode's format isn't optimized for searching, simply a linear list 
of name/value pairs - no btree, no hashes, none of the goodies that make 
them so fast in RAM.

Interestingly, some benchmarks I did some time ago suggest that the 
format for custom properties is at least somewhat hashed, so that 
accessing a single property tSomeKey in a property set uProp like this:

   get the uProp[tSomeKey] of this stack

...is much faster than obtaining the same value if you'd stored the 
array as a single property like this:

   put the uProp of this stack into tArray
   get tArray[tSomeKey]

In fact, the overhead of arrayDecode is so significant, and the 
optmization of custom property access so nice, that in many cases you 
can access a series of custom properties faster than you can access a 
series of array elements from a variable if you must first obtain the 
array from any other persistent store (either property or arrayEncoded 
file).


 > Either way my guess is you are having to do many iterations to see
 > a measurable difference unless you are dealing with an amount of
 > data that would be better in a database... so no matter what the
 > engine needs to do the reason why we can't deal with multi-
 > dimensional custom properties is because of the syntax ambiguity.

If we were as casual about all optimization opportunities, we might 
parse text runs in fields rather than in htmlText. ;)

Not all data stores need be relational, and it's possible to craft 
persistent stores in LC that outperform SQLite if your needs are modest 
and performance is important for that particular use case.

So respectfully, the parser and the storage format are two separate 
challenges, and both would need to be addressed to make good use of 
nested arrays as persistent stores.


 > The simplest resolution to the ambiguity is to only allow array
 > syntax on sets so the set name must be the first key. If you need
 > to access one of the regular custom properties there is already a
 > mechanism in place for that by using empty as the set name.
 >
 > get the customKeys[""]["dgData"]["1"]
 >
 > get the customKeys["dgProps"]["row template"]

I'd vote for that.

And with that, how about a btree-based storage format so we could do 
that from disk as well. :)

--
  Richard Gaskin
  Fourth World
  LiveCode training and consulting: http://www.fourthworld.com
  Webzine for LiveCode developers: http://www.LiveCodeJournal.com
  Follow me on Twitter:  http://twitter.com/FourthWorldSys