Quit Command corrupts standalone (stack called by standalone splash)

[Richard Gaskin]

Knapp Martin wrote:

 > Thanks for the in-depth reply. Are there known issues with saving LC
 > stacks to Dropbox? I was thinking of Roland's issue with corruption
 > and Phil's suggested fix of watching the file with the tilde character
 > before allowing a quit to complete. I have an app where users create
 > documents (stacks) and saving them to Dropbox is a built-in option.

My hunch (read, "I'm likely wrong, wait for something more definitive 
from someone on the core team or Brian Milby, since he reads the engine 
source <g>) is that LC files are less prone to a wide range of 
disk-related issues by virtue of being written in a single pass.

HyperCard, FileMaker, SQLite, and others are read and written in pages, 
which works more reliably in some (SQLite) than others (no one misses 
HC's 504 errors <g>).  Moving stuff in and out of portions of a file 
that may not change much otherwise isn't necessarily brittle in itself 
(SQLite as a reputation for being enormously robust), but we can 
envision how larger SQLite files may confuse a syncing system that can 
only move chunks of data over a network so quickly.

Since LC files are, near as I recall, written fresh from beginning to 
end with each save command, the result is usually a cleaner file than 
with some paging systems.

And it's also faster to write everything in one pass than writing the 
same amount of data by moving back and forth throughout a file, so my 
hunch would be that LC stack files would likely be less prone to error 
when synced over Dropbox than SQLite files.

But by no means should this be taken as gospel, and certainly not as any 
indication that we should drop SQLite and use stack files instead. 
Different formats for different purposes.

And as Mark Wieder noted, some of the biggest challenges with syncing 
occur with multiple users accessing a given resource simultaneously.

Those are very real challenges, which may be why so many developers over 
the years keep writing their own sync mechanisms, with no single 
solution having emerged as the one-size-fits-all answer.

Consider this scenario with stack files, for example:

Mark writes a stack, then I open it, then you open it.  You and I are 
both making changes, and I save mine a few seconds before you save 
yours. In that scenario, what's on disk?

When I get back to work on the file tomorrow, I'll likely be 
disappointed to find that everything I'd done is now gone.  I can see 
only your changes, since yours were the last saved and when you saved 
you hadn't yet updated to have a copy with my changes.  So after Dropbox 
faithfully performs everything we ask it to do, my changes are gone forever.

Write-locking can help with things like that, and that's a subject in 
itself.  At what level of granularity should we lock?  And before 
editing begins do we first compare what's in memory with what's in the 
data store to ensure what we're about to edit is the most recent copy? 
How do we handle conflicts, when one inevitably occurs?

So many ways to handle sync....

 > I have one customer who stores their stacks on a network server and
 > a few times they've ended up with a corrupted stack.

Corruption is very interesting to me.  Lost data is usually explainable 
through simple multiuser workflow weaknesses, such as inadequate locking 
mechanisms.  But corruption in LC means something interrupted the write, 
or overwrote portions before the write was completed.  And since LC 
stacks files are written in a single pass, whatever caused it to corrupt 
was very efficient at it, as it had little time to do it in.

I can conceive of a possible scenario in which neither Dropbox nor LC 
has a bug per se at all, but which may account for corruption:

1. Your app issues a save command on a large stack.
2. Dropbox begins syncing the file to the server.
3. Your friend has the same file open on his machine.
4. On his machine, Dropbox notices the server has a newer version, and 
prepares to update his copy.
5. He starts a save in his stack.
6. Dropbox has written a portion of the stack from the server to his 
disk, but stopped when it noticed the file was being updated.
7. After your friend's save is done, the corrupted stack is then copied 
to the server, where it later propagates to your machine.
RESULT: everyone now has the same file, and it's broken.

I haven't studied the file system triggers and server monitoring that 
trigger reads and writes in Dropbox.  And if it's proprietary, chances 
are no one outside that company has either.

There are many ways that imagined scenario might for all I know be 
accounted for in the way Dropbox is written.  But there may also be 
other scenarios that can produce the same corrupted result that I 
haven't thought of.

-- 
  Richard Gaskin
  Fourth World Systems
  Software Design and Development for the Desktop, Mobile, and the Web
  ____________________________________________________________________
  Ambassador at FourthWorld.com                http://www.FourthWorld.com