Arrays vs databases

[Richard Gaskin]

Matt Maier wrote:

 > According to this lesson, I can store an array as a custom property
 > of a stack which can be saved and loaded.
 > 
http://lessons.livecode.com/m/4071/l/17375-how-do-i-save-custom-properties-in-a-standalone-application
 >
 > Something I've done myself already is to either save the array as
 > JSON in a text file or just as a binary with encode.
 >
 > So, in the most naive sense, it looks like I can have a key-value
 > store without installing and learning anything new. But databases
 > have extra functionality to deal with stuff like multiple
 > simultaneous users, data integrity, and performance optimization.
 >
 > Can I just setup an instance of Livecode to act like an in-memory,
 > key-value database? Maybe have a standalone stack act as the
 > controller and have it mirror it's state in a separate stack that
 > can be saved and loaded.
 >
 > I don't know enough about database design to frame the tradeoffs. Can
 > someone who groks both Livecode and databases comment on this idea?

I think you're on a very useful path.

TL/DR version:

LiveCode's encoded arrays are a good fit for many key-value storage needs.


Typically windy version:

RDBMSes are great when you have data too large to fit into RAM, or when 
the nature of the data or its workflow lends itself to being expressed 
through normalized relationality.

But for simple key-value stores they can be overkill.  Simple arrays 
work very well for those, but we need some form of persistence.

JSON is usually the go-to solution for serializing arrays, and it's 
necessary to use either it or XML when transferring data between systems 
that know little about each other, since both are widely-used standards.

But when data will only be used within LiveCode we have much broader 
options.

I used to use stack files for data storage, but I've come to find them 
cumbersome for two reasons:

1. Purging them from memory requires careful thinking (we have no "purge 
stack" command, and "delete stack" is disconcerting to use because it 
can only safely be used on mainstacks, but using it on a substack will 
do what it says on the tin and delete the stack object from the stack file).

2. Given property sets and properties, array notation can only be used 
natively on object properties with two levels of depth.  It's possible 
to store entire arrays as property values, but those require that you 
first extract them into a variable, and if you're going to do that you 
might as well bypass the concerns of #1 by using:


My favorite key-value format is LSON, my pet name for encoded LiveCode 
arrays.

I call them LSON because everyone's familiar with JSON, some are 
familiar with MongoDB's BSON ("Binary JSON"), and LiveCode's encoded 
arrays are similar enough in structure and use case with BSON to merit a 
similar label.  I find when I use anything ending with "SON" 
conversations with clients become much easier; they grasp the concept 
and benefit immediately.

You can serialize an array with arrayEncode, and de-serialize it back 
into an array with arrayDecode.  Once serialized, it can be used like 
any other data, saved to disk or sent over a network.

JSON is popular because JavaScript is popular, and JSON was designed to 
be de-serialized with a single function call in the JS engine, eval.

But every language that isn't JavaScript has to go through the 
cumbersome process of parsing it, notoriously slow compared to the 
optimized handling provided by JavaScript's machine-code-compiled 
engine. JSON is a glove that perfectly fits the hand of the JavaScript 
engine, but any other engine needs to do quite a bit of work to emulate 
what the JS engine does so easily with it.

So when data is to be used within LiveCode, LSON is by far the better 
choice.  It's functionally equivalent to JSON not only terms of 
serializing hierarchical data, and also in terms of the performance and 
robustnesss of being able to rely on a single engine 
machine-code-compiled function to serialize and de-serialize it.  LSON 
is a glove that perfectly fits the hand of the LiveCode engine.


Extra bonus points:

Key-value stores needn't be limited by RAM.

Studying MongoDB, CouchDB, and other large-scale key-value stores can be 
quite inspiring.

Of course if your collection is really large or intended for use by many 
hundreds of concurrent users, you're probably better off just using 
MongoDB or CouchDB. Greg Miller has a nice CouchDB library for LiveCode 
on Github (background an detail in this forum thread:
<http://forums.livecode.com/viewtopic.php?f=12&t=24925>).

But for workgroup solutions, in which the number of concurrent users may 
be only a few dozen and the collection size below several tens of 
thousands, it's possible to build reasonably performant client-server 
solutions using LSON as the storage format, even with the limitations of 
CGI.

And if you're running a VPS or dedicated server the scale can be much 
larger, since you're no longer bound by the CPU and RAM constraints 
imposed by CGI.  The MongoDB daemon, for example, keeps its index in 
RAM, used to map access to the much large disk store.  This can be done 
with a LiveCode-based daemon as well.  The experiments I've done along 
those lines have gone very well, and I believe the backend for Canella 
Software's LiveCloud works similarly.

Using LiveCode-native LSON for key-value serialization opens many doors 
for data storage solutions, as efficiently as JSON does for 
JavaScript-based systems.

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com