Word chunk includes punctuation

[Paul Dupuis]

The documentation also doesn't mention that ? works as a token per
Mark's post, so perhaps my caution is unwarranted.

Silly me, I believed the documentation!

On 8/13/2012 5:01 PM, Bob Sneidar wrote:
> Hmmm... odd again that 
> put token one of word one of "test1!"
>
> gives me
> "test1" without the exclamation mark. 
>
> The documentation states:
> 1. Each of the following characters is a token: =, +, -, *, /, [, ], (, ), {, }, <, >, and comma (,).
>
> The exclamation mark is not one of those (unless I am token blind [again]) and yet it seems that it is being treated as one. 
>
> This looks very much like an error in documentation. 
>
> Bob
>
>
> On Aug 13, 2012, at 1:32 PM, Paul Dupuis wrote:
>
>> One caution: token does not separate . (period), ! (exclamation mark),
>> or ? (question mark). If you are really trying to process English text,
>> you probably will want to write your own punctuation remover as it can
>> then figure the difference between a period at the end of a sentence and
>> a period at the end of abbreviations like "Dr." or "Mr."
>>
>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>

-- 
Paul Dupuis
Cofounder
Researchware, Inc.
http://www.researchware.com/
http://www.twitter.com/researchware
http://www.facebook.com/researchware
http://www.linkedin.com/company/researchware-inc