Stress-testing SQLite -- millions records? Use Valentina DB

[Ruslan Zasukhin]

On 10/30/10 3:17 AM, "Richard Gaskin" <ambassador at fourthworld.com> wrote:

Hi Richard,

> I have a need coming up for a data store that can robustly handle at
> least a million records, ideally up to five million, where each record
> may be as large as 5k.
> 
> I don't need relationality, so for me SQLite is an option but only an
> option; I'm happy to consider other options as well. (Yes, it has to be
> SQLite rather than MySQL, because it needs to work embedded with a
> commercial application).

Valentina DB is faster 100 times of SqlLite, mySQL

And can be perfectly embedded into commercial application because it is
royalty free.

 
> Have any of you done stress testing on SQLite to that degree?
> 
> I've tried finding even anecdotal data on the web for SQLite limits, and
> while I can find citations of theoretical limits I haven't come across
> real-world usage stories of data sets that large.
> 
> Should I be confident in SQLite as a storage solution for that?  Should
> I be scared?

SCARED :)
 
> Thanks in advance for sharing your experiences with large data sets if
> SQLite.

You need Valentina DB.

Okay you need 5K for each record.
How many fields?

Let me remind that Valentina has columnar format.
This is huge advantage.

Also Valentina can give you not only SQL way but NON-SQL way,
Which can be additionally 10-20 times faster!

----------
I can tell you store, that Valentina was tested for AOL Europe by their dev
team. Against Berkly,  mySQL, postgre, and other dbs. SqlLite even was not
in game of course.

Task was so simple. Table has 2 fields {URL, PictureBannerAd }

So when somebody ask for a WEB page, it needs find banner to be shown.

As they told, e.g. Berkly have give 100 faults per time (min our hour I not
remember now).  Other dbs also. Fault means that banner was not found  by DB
in time less of timeout.

Valentina have give them zero faults.


Let me underline this very important feature NON-SQL-ness of Valentina.
As well as very powerful SQL.

Today is very modern stream talk about how SQL DBs are bad, and how cool are
NON-SQL with Key-Value.  Guys, be happy, Valentina is perfect for both
tasks. :-)

If talk about details, in V4REV API (and most others Valentina ADKs) you can
use not SQL way to do searches and sortngs using
   VField_FindValue()
   VField_FindRange()
   VField_FindLike()

And other similar search methods.  They are really FASTEST POSIBLE way.
 

-- 
Best regards,

Ruslan Zasukhin
VP Engineering and New Technology
Paradigma Software, Inc

Valentina - Joining Worlds of Information
http://www.paradigmasoft.com

[I feel the need: the need for speed]