custom protocol for http large data (was Re: Maximum size of POSTdata)

[jbv]

Andre,

> JB,
>
> nothing beats experience. Could you tell us more about your protocol,
> it's design, how did it work and the like, I am very interested in
> anything related to this.

Well, it was a year ago, so I may not recall every little detail...
Basically, the web site was made of different sections, each section
featuring various pages that had to be updated or added several times
a day. Each page was built according to a specific template / layout,
but all
pages contained roughly the same kind of elements :
- 1 large photo (100 / 150 Kb)
- 4 to 10 small photos (30 / 60 Kb)
- legends for the photos (60 chars)
- some text : title (100 chars max), introduction (500 chars), article
(2000 /
3000 chars).

The update was made by someone from the middle of the moroccan
desert, with a laptop hooked to a cellular phone with a satellite
connexion.
Due to the cost of bandwidth, posting data had to be fast & efficient.

So I built a standalone with MC. It featured as many cards as sections &

layouts, each card reproducing the specific layout (with imgs, text
flds,
etc), and also tools to link photos & text to each element of the
layout,
to know the number of chars, to check if any element was missing, etc.
Then the user would click on a "send" btn, the app would open the
connection
and the content of the displayed card would be posted to the server.

As for the protocol, clicking the "send" btn would start a main loop,
sending
each element after another, and waiting for confirmation from the server

before sending the next element, or re-sending the same if anything went
wrong.

AFAIR, for photos, various parameters were included in the post, namely
:
- number of parameters to follow
- section name
- date
- format (gif or jpg)
- main (or small-1, small-2...)
- legend
- number of bytes to follow
- raw bytes of the photo

For text, it was a bit simpler :
- number of parameters to follow
- section name
- date
- format (text)
- number of bytes to follow
- raw bytes of title
- number of bytes to follow
- raw bytes of introduction
- number of bytes to follow
- raw bytes of article

When receiving data, the cgi script on the server would name and save
each photo
in a specific directory (according to the "section" and "date"
parameters), and would
build and save the html page (containing the text and the links to the
photos) according
to the same parameters. Meanwhile it would also update some links in a
general menu
when new pages were added.

IIRC I was able to complete that app in about 1.5 day. Testing it was
rather simple, as
the protocol was the same for each element of the page.

OTOH I remember spending 2 or 3 days on a cgi script handling multipart
data (mostly
text and photos that end users were able to upload to a database). I
know that there are
plenty of ready-made PHP and Perl scripts to handle multipart data from
html forms,
but all cgi scripts on this project were made with MC, so I decided to
keep everything
consistent.
And even if I managed to finalize a script tailored to my needs, I would
certainly not
share it to anyone, as it's far from a general purpose solution...

Hope I was clear...
JB