Keys of a custom property array

[Monte Goulding]

On 05/03/2013, at 10:56 AM, Richard Gaskin wrote:

> > I agree it makes total sense to use the same format when
> > reading/writing to disk. As all the data is loaded into memory anyway
> > I wonder how much extra time it would take to turn it into a pointer
> > that was just as fast to access as a regular array.... perhaps that
> > could be done at first access?
> 
> ArrayEncode's format isn't optimized for searching, simply a linear list of name/value pairs - no btree, no hashes, none of the goodies that make them so fast in RAM.
> 
> Interestingly, some benchmarks I did some time ago suggest that the format for custom properties is at least somewhat hashed, so that accessing a single property tSomeKey in a property set uProp like this:
> 
>  get the uProp[tSomeKey] of this stack
> 
> ...is much faster than obtaining the same value if you'd stored the array as a single property like this:
> 
>  put the uProp of this stack into tArray
>  get tArray[tSomeKey]
> 
> In fact, the overhead of arrayDecode is so significant, and the optmization of custom property access so nice, that in many cases you can access a series of custom properties faster than you can access a series of array elements from a variable if you must first obtain the array from any other persistent store (either property or arrayEncoded file).

Hmm... I think your test methods might need to include loading the stackfile into memory otherwise you are comparing data in memory with reading data from disk...
> 
> 
> > Either way my guess is you are having to do many iterations to see
> > a measurable difference unless you are dealing with an amount of
> > data that would be better in a database... so no matter what the
> > engine needs to do the reason why we can't deal with multi-
> > dimensional custom properties is because of the syntax ambiguity.
> 
> If we were as casual about all optimization opportunities, we might parse text runs in fields rather than in htmlText. ;)

I'm not being casual.. I definitely think there's good reasons to optimise this... See my idea above where the custom property would be just as fast as working with an array variable once in memory.
> 
> Not all data stores need be relational, and it's possible to craft persistent stores in LC that outperform SQLite if your needs are modest and performance is important for that particular use case.
> 
> So respectfully, the parser and the storage format are two separate challenges, and both would need to be addressed to make good use of nested arrays as persistent stores.

Hmm.. there are many potential storage formats for an array. It would be nice to have some level abstraction that allowed for extensions.. in open language syntax:

encode <array> as JSON | XML | btree | ....

Cheers

--
M E R Goulding 
Software development services
Bespoke application development for vertical markets

mergExt - There's an external for that!