Word chunk includes punctuation

Mark Wieder mwieder at ahsoftware.net
Mon Aug 13 16:50:06 EDT 2012


Monday, August 13, 2012, 1:32:58 PM, you wrote:

> One caution: token does not separate . (period), ! (exclamation mark),
> or ? (question mark). If you are really trying to process English text,
> you probably will want to write your own punctuation remover as it can
> then figure the difference between a period at the end of a sentence and
> a period at the end of abbreviations like "Dr." or "Mr."

Good point. A question mark does count as a word separator, so "token
1 of word 1 of..." will still work, but the other two could cause

-Mark Wieder
 mwieder at ahsoftware.net

More information about the use-livecode mailing list