Using stacks as a database for very large data sets

[MisterX]

> Saving a large stack can be time consuming - especially 
> across a network.

I've complained about this. But the problem was in the
way I worked. OK, the stacks are not rewritten in the 
same way as they did in HyperCard - but it's unlikely
that's going to change anytime. 

Even though most of my stacks are small some production stacks go up to
10megs size.

Forget about using/saving those across the network - it's asking
for trouble. On windows, if there's a net outage - it's a crash most
probably! But you can save the stack on some occasions before!

You should try our gigabit network to save a 1mb stack! It takes up to 20-30
seconds!

Nothing is slower thanks to firewalls and security checks!
Welcome to the enterprise secured network! But it's either that or work on a
thin client! NOWAY! 

What i found is these methods:
I've reduced stack size to a minimum. 

Work locally for development and then copy back to the slower network where
the stack is used. 

This allows you keep working while the development copy is being updated if
you find a bug!

If you have to work with the stack opened on the other side (one user only
always) then you should work with merging separate data files in text, xml
or small update stacks with extra cards to update the working database. 

Although that means that you could also send the information via network
with sockets to the other side - if needed - it's not as easy or solid imoho
as the stack save / extract/merge approach. With a good protocol code it
could be done though. But then you need firewall clearance! ahem...

here's the pseudo code to send a stack or post card file the easy route!

clone stack StoryUpgrade
repeat for each newstory x
 create card 
 put story[x] into fld "data"
end repeat 
save stack StoryUpgrade
copyfile fromthispath to thatpath
sendmsg "netrevipname:thisport", "updatestories"

voila!

The other side receives the message...

on updatestories
  set the defaultstack to updatestories
  repeat with x = 1 to the number of cards
    put fld 1 of card x into story[laststory+x]
  end repeat 
  add x to laststory
  repeat with xin = 1 to x
    create card
    put story[xin] into fld 1 of it
  end repeat

It's pseudoscript but it's how i see it coming soon somewhere in my stacks!
I think it's safer instead of sending the data because it sends a file
instead of fragments of data which i've seen often fail - more often than
file transfers. File transfers can be easily resumed, checked, waited for. 

Naturally, an sql database available across the network would be probably
better in the long run! But commiting some data or commands can be dangerous
without a test database first! That's why i like the simple stack approach.
You can pack as much as you want in any stack - I've done 600000 records
across 6 levels with referencing - it's not hard, it's quick and works
reliably.

Also I think that in many cases (dev -> production) sending data via sql is
slower than via file network transfer. So if you send it as a stack to the
machine that inputs it directly/locally on the sql db, you got a shortest
route possible! Depends on the frequency you need to update. 
You can also compress data in a stack to reduce file size...

OK, enough computers and "you can alsos" for me tonite! I must prepare for
my first snowboard ride with a semi-healed broken heel-bone that's F1
weather predicting sensitive! I got the new snoboots and couldn't go riding
xmas time, missed saturday's compromised weather (turned out superb), worked
sunday (superb day too) but managed to get the day off to ride tomorow
across "les Vosges" not far from Strasbourg.fr! They got 3 days of non stop
powder falling and nobody on the slopes!!! it's ideal as a clean stack!

revgourl "http://google.com "
type "la bresse webcam"
send "mouseup" to button "im feeling lucky"

sorry for the lengthhhh, i had to get it out! Hope it helps nonetheless

Xavier
--
monsieurx.com - So what? Im having fun!


> -----Original Message-----
> From: use-revolution-bounces at lists.runrev.com 
> [mailto:use-revolution-bounces at lists.runrev.com] On Behalf Of 
> Lynch, Jonathan
> Sent: Wednesday, January 26, 2005 16:53
> To: How to use Revolution
> Subject: Using stacks as a database for very large data sets
> 
> 
> Saving a large stack can be time consuming - especially 
> across a network.
> 
> Would it be possible to have some sort of write-to-file 
> scheme that just changes the stored data for single card, but 
> not the entire stack?
>