Word chunk includes punctuation

[Peter Haworth]

I've used tokens quite a bit for parsing sqlite statements and the docs are
definitely incomplete.  I'll try to rebuild what I learned and publish it
to the list.
Pete
lcSQL Software <http://www.lcsql.com>



On Mon, Aug 13, 2012 at 2:01 PM, Bob Sneidar <bobs at twft.com> wrote:

> Hmmm... odd again that
> put token one of word one of "test1!"
>
> gives me
> "test1" without the exclamation mark.
>
> The documentation states:
> 1. Each of the following characters is a token: =, +, -, *, /, [, ], (, ),
> {, }, <, >, and comma (,).
>
> The exclamation mark is not one of those (unless I am token blind [again])
> and yet it seems that it is being treated as one.
>
> This looks very much like an error in documentation.
>
> Bob
>
>
> On Aug 13, 2012, at 1:32 PM, Paul Dupuis wrote:
>
> > One caution: token does not separate . (period), ! (exclamation mark),
> > or ? (question mark). If you are really trying to process English text,
> > you probably will want to write your own punctuation remover as it can
> > then figure the difference between a period at the end of a sentence and
> > a period at the end of abbreviations like "Dr." or "Mr."
> >
>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>