Array literals and script only stack initialization (was Re: Developing first on android)

[Mark Waddingham]

On 2017-08-22 00:23, Monte Goulding via use-livecode wrote:
> I can’t recall getting an explanation of what the dead end was but my
> guess is that it was fact that a merge conflict would be a nightmare
> to sort out and it is _very_ hard to write a non-lossy stack file
> format that won’t have a lot of merge conflicts

I think we shelved that project before you joined LiveCode :)

The 'dead-end' was more a realisation of the fact that we had been 
trying to solve the wrong problem.

All the dVCSs which exist today are text-based - they are designed and 
built for allowing collaborative contribution to projects which are 
*naturally* text-based. Indeed, git was designed by Linus Torvalds to 
improve efficiency of contribution to the linux kernel - which is a 
purely C project.

When you combine this with an extra constraint which was placed on the 
project - the ability to be able to 'naturally' interact with patches 
through GitHub or similar - you end up with the dead-end.

The goal of trying to create a mergeable text format for stacks which 
should work naturally through GitHub and such was, I think, a mis-step. 
It just isn't possible, in full generality, to do in a way which does 
not involve significant amount of human effort to resolve conflicts at 
the text level (and conflict resolution can be hard - even in C and 
similar languages - for a hierarchical data-structure it can be a lot 
harder!). This means that merges require a tool, and if merges require a 
tool why do we care about the underlying format?

The point here is that, with hindsight, we would have been better off 
ignoring the requirement for things to work 'in GitHub', and instead 
producing a merge-tool which could take two stacks (which had been 
instrumented appropriately) and produce a third after guided interaction 
to resolve the conflicts. Of course, text display of stackfile content 
would be useful in this - to help visualise the conflicts - but it 
wouldn't need to be total, or complete, or satisfy some impossible 
constraints around ensuring merge conflicts (detected with 3-way diff) 
were minimized.

Morale of the story: don't expect an algorithm (diff) which is designed 
for giving you the differences for sequential data to work on 
hierarchical data - because it won't.

> If the goal of any array import/export of LC object is to create a
> mergeable file format then I wouldn’t bother. There’s just too much
> mingling of data and session state in LC objects. lcVCS just barely
> scrapes by if you have rigorous object cleaning scripts so that you
> don’t get merge conflicts on stuff like object sizes in a resizable
> stack. Not to mention the object ID merge conflict conundrum which
> needs a _lot_ of code to work around the fact we don’t have UUIDs.

 From my point of view, array representation of core engine object's 
properties is a useful feature to have. It can be done now (lcVCS shows 
that), however, the code which does it is spread around several places 
which means maintenance is difficult. The thing which generates the 
array format should be a method of the object which produces it.

I'd be wary of adding anything beyond *just* the core properties array 
export to the C++ part of the engine as the overhead in writing the C++ 
to do more than that is not justified - the construction of a full array 
of a stackfile would be better done as a LiveCode Script library. If 
there turn out to be performance issues with that approach which were 
critical *then* we look at how to make the script more performant, 
perhaps by adding engine functionality for the parts causing a problem.

> With script only stacks proving there’s significant utility in lossy
> file formats here I think the best solution would be something along
> the lines of the script only UI library I made where only a very
> limited subset of properties is supported, everything _must_ be
> uniquely named, no custom properties, no non-stack behaviors, only
> store the name of images used for icons etc. Just the absolute bare
> minimum to recreate the stack. Use something like YAML so it’s super
> easy to read and make it a single file so it’s not possible to get
> lost in an arcane directory structure.

The reason Levure works well is because it plays by the implicit rules 
which you need to follow if you want to use GitHub-like dVCS - your 
project must be a naturally text-based format, where naturally means 
designed to be written in text directly rather than rendered to text 
using some sort of algorithm. Indeed, more specifically, it also needs 
to follow implicit rules which ensure conflicts are minimised - one 
being unique naming of objects!

Certainly YAML would make for quite a readable format - although it is 
unlikely it would be any better or worse, in terms of mergeability, than 
any of the other approaches that have been tried (unless generated in a 
specific canonical way, and the stacks rendered to it follow some very 
very strict rules - which is essentially, then, something like Levure).

Warmest Regards,

Mark.

-- 
Mark Waddingham ~ mark at livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps