Using stacks as a database for very large data sets

[Gordon Tillman]

Hi Jonathan,

On Jan 26, 2005, at 11:01, Lynch, Jonathan wrote:

> Still - how do database programs do it? Is each record a separate file
> or something? If it were possible to use a single stack as a huge
> database, it would be very convenient.

The big problem of course is that opening a stack loads the entire 
thing into memory.  With a real database, certain data-structures are 
maintained in memory -- and the sizes of those are usually tunable for 
performance -- but the database files are not all loaded into memory at 
once.    For the most part each record is not a separate file.  What is 
typically done is something like this:  The database files maintain an 
internal fixed block size.  When records are written to the file they 
can span multiple, discontinuous blocks.  The blocks for deleted 
records can be reclaimed and reused by new or updated records.  Indexes 
are maintained (sometimes in separate files, sometimes in the same 
file) that allow for fast lookup of records in the database.

--gordon