Using stacks as a database for very large data sets

[Rob Cozens]

Hi Bruce,

 From my conversation with Jonathan:

 >>
>Saving a large stack can be time consuming - especially across a
>network.

One alternative is a client-server setup.

The entire stack is in RAM on the server side; but client stacks only deal 
with one record at a time.

Thus the stack is always saved locally at the server, never over the network.

>If it were possible to use a single stack as a huge
>database, it would be very convenient.

I've worked with 44K-record, 43MB, database stacks in 
SDB.  <http://wecode.org/serendipity/>

> >> Maybe for Rev 3.0 they will work out such a thing
> >
> > If they don't, I have--if you're ready to go client/sererver

<<


>Well, that sounds like the right approach. Brief summary?

Serendipity Database--Binary (SDB) is a hierarchical database scripted 
entirely in Transcript.  It is implemented as a library stack containing 
database & general handlers, icon images, dialog substacks, and a generic 
database template substack.  SDB ships with a reference stack & back-end 
database, several tutorial/test stacks, a developers' plugin stack, and 
template stacks for creating SDB Utilities & SDB Server standalone 
applications.  SDB single- and multi-user syntax are identical, and 
database front end stacks/standalones can switch between the two modes 
during runtime.  When operating in client/server mode SDB uses the 
Revolution IPC Group's Lib_IPC & Lib_STAMP TCP/IP handlers for 
inter-program communication.  SDB supports explicit locking at the database 
& record level and an optional "auto lock" mode where each user's 
currently-selected record is locked until the user logs off or changes 
current database position.  SDB is 100% open source, modifiable, and 
royalty-free.

When data must be shared among several users, a client/server approach 
eliminates the need for application logic to avoid data collision: only the 
server app has access to the database stack, and it responds to client 
requests on a FIFO basis.  In a Transcript environment, this also has the 
advantage that the entire database stack only resides in RAM and client 
applications only buffer one record at a time.

Performance with large databases is directly dependent on server RAM and 
clock speed.  SDB's binary index searching is able to retrieve one record 
out of 44,000 (43 MB) in < 1 second on a midline G4 iMac.  Times to 
load/save a large database will show the same correlation with hardware 
speed and capacity,

Rob Cozens CCW
Serendipity Software Company

"And I, which was two fooles, do so grow three;
  Who are a little wise, the best fooles bee."

  from "The Triple Foole" by John Donne (1572-1631)