Data synchronization

[Alex Shaw]

Hi Andre

Thanks for the tips!

I've been working on a similar idea & you have helped assure me I'm on 
the right track :)

Andre Garzia wrote:
> Hello Alex,
> 
> I don't know much about Valentina but I decided to chop in this thread
> (which I read and learned much from). If you make a litte server using
> Valentina and an HTTP server on front, I'd advise you to work the following
> way:
> 
> Define a clear HTTP based API for accessing your server. Try to use whats
> being called "REST" which is a fancy name for using URLs and HTTP Methods to
> simplify interfacing with a server, it's way easier to work with a RESTful
> server than working with SOAP.
> 
> script your GET requests to acquire data from the DB. You can make your URLs
> match your database design. for example, you're building a documentation
> database:
> 
>   GET  /  (gets all records)
>   GET  /category  (gets the categories available)
>   GET /category/installing  (get records with category "installing")
> 

I agree it would be easy to just create a custom protocol using sockets 
but using URL/XPath style namespaces is more flexible and can be 
utilised down the track for other types of clients (ie web browsers, 
flash etc).

> The idea is to make sane URLs that are both human readable and machine
> processable. If your entries have a timestamp marking the last modification,
> you can create a URL such as:
> 
>   GET /recentchanges  (get the newest records)
>   GET /modified/2007/10/06  (get records modified since 2007/10/06)
> 
> This will make easier to detect changes in the DB and replicate the changes
> between clients. Use PUT calls to insert data onto your database, you can
> put the content of the request as a XML with the fields and data for your
> DB. If you're felling bold, you can use smart urls for insertion too.
> 
>   PUT /  (all content for the record is inside the request)
>   PUT /category/installing  (the "category" for this request is obtained
> from the URL)
> 
> The above snippet is not as useful as the GET requests, inserting data is
> easy, querying data is what makes your system easy to extend and implement.
> To modify a record use the POST method. Here it is a good idea to use fancy
> urls such as:
> 
>   POST /id/1234 (changes record #1234)
> 
> Or you can insert all data into the request. The idea of URLs for modifying
> data is to be able to query and modify batches of records with a single
> request. For example suppose your documentation system has entries tagged as
> "draft" or "publish", by switching this status setting you make some FAQ
> available to the end user. You can have a request such as:
> 
>   POST /drafts/user/Soapdog  (will change all drafts by user "soapdog")
> 
> In this request you can simply add the XML to update the draft field to
> publish. This can easily be done with SQL queries and in the end, this will
> of course become a SQL query somewhere inside the system. The question you
> must ask yourself is, do I want to put SQL in my HTTP Requests or should I
> build my own XML/URL Dialect to make queries?

The interface allows "admins" to attach SQL statements to objects but I 
plan to move away from this & parse XML control statements.. with the 
smart custom http server interpreting smart urls.

> 
> The second takes time but pays in the long run since you can tailor your
> software to the specific needs of your system/workflow. This option will
> make you have the better tool for doing your job since you created a DSL
> (domain specific language) that is fine tunned to your job instead of using
> the general purpose SQL.
> 
> Now getting back to the syncronization problem. Using the smart URLs you can
> discover what's been changed in the database and fetch them easily. By
> fetching just what you need, you reduce your bandwidth usage and you also
> reduce the load on the RevOnRockets server which is single threaded anyway.
> You can also make a little MD5 field on your records that you can use as a
> locking sytem so that users don't overwrite each other. This MD5 record must
> be sent with each POST request, if they don't match the one in the central
> database, it means someone else altered the records in the meanwhile and you
> should refetch that record and revise your changes.
> 
> does this help?
> andre
> 

Yes, again thanks Andre, it does indeed help in many ways.

I'm still finalising table structures & basically plan to use the 
relational database in an object-orientated way with rev looking after 
final viewing structure. Ignoring the boos & hisses from the "serious" 
database people, who seem to emphasise that not only should data be 
stored in the database but also a certain level of function..

Fortunately we have a great tool called "Revolution" that makes this 
possible.. well, it works for me and certainly this project.

User can ultimately only change one bit of information at a time so I 
plan to embed another customised rev http server within the client apps 
so that the main management server can push updates in real-time to each 
client. It can then send updates down a chain of backup/replication 
servers if need be.

The synchronization should now be quite flexible based on a few simple 
rules. ie. user conflicts are overridden by main server but admins can 
choose from a number of options.

Instead of MD5 I will probably use a SHA1/HMAC approach (nice 
implementation Mark!) so all communication messages can be authenticated.

Certain data will be encrypted locally but can go either way on whether 
to encrypt comms on-the-fly or use vpn (which is probably the easier 
option & more flexible). Currently larger files are just send via ftp, 
with all machines running a ftp server, and the custom management server 
sorting things out after transfers are complete. Would be nice to 
implement a custom embedded ftp server (to keep things under the same 
roof) but would rather concentrate on other more important issues.

Anyway, see how we go ;)


regards
alex