Handling of final delimter (was Re: "this me"?)

[Richard Gaskin]

Mark Wieder wrote:

> "1,2,3" -- this contains three items
> "1,2,3," -- this contains three items
> "1,2,3, " -- this contains four items

Let's look at those strings from the point of view of what we might 
presume was that of the HyperTalk design team, but besting them by using 
Jacque's suggestion of the more appropriate "terminator" rather than the 
ambiguous "delimiter" - here we'll use "x" for any data that isn't a 
terminator, and "|" for "terminator:

x|x|x -- this contains three items
x|x|x| -- this contains three items
x|x|x|x  -- this contains four items

Once we understand that delimiters are actually terminators, even empty 
items become easily graspable:

x|<empty>|x|x - this contains four items

...because what we're counting is any chunk that's terminated, either by 
the delimiter or by simply being the end of the string, even those 
elements whose value is empty.

One simple solution would be to add "lineTerminator" and 
"itemTerminator" as synonyms for "lineDelimiter" and "itemDelimiter" 
respectively, and then the conceptual model fits what the engine does. 
We leave the "*delimter" tokens in place for us ol' timers, but 
emphasize "*terminator" for newcomers.

A more complex solution was hinted at in a recent post by Klaus in the 
forum thread on this, in which apparently Mark Waddingham has been 
giving this a good deal of thought and has come up with a solution that 
obviates the issue altogether by introducing true lists, rather than 
just delimited strings, so there are no delimiter characters per se.  I 
believe Lingo is one xTalk derivative that supported true lists, and 
many other VHLLs offer them too.

Personally, I'd like to see both implemented, as each can be very useful 
for different tasks.

If we get both of those (more accurately descriptive tokens and true 
lists), once folks get a chance to use them we can then evaluate the 
usefulness of bifurcating the community code base along the lines of 
those handlers which expect traditional terminators and those which use 
the new delimiter behavior dependent on a global setting.  But I'd wager 
that if we had both very few scripters would be motivated to pursue this 
further. :)

--
  Richard Gaskin
  Fourth World
  LiveCode training and consulting: http://www.fourthworld.com
  Webzine for LiveCode developers: http://www.LiveCodeJournal.com
  Follow me on Twitter:  http://twitter.com/FourthWorldSys