Word chunk includes punctuation

[Mark Wieder]

Paul-

Monday, August 13, 2012, 1:32:58 PM, you wrote:

> One caution: token does not separate . (period), ! (exclamation mark),
> or ? (question mark). If you are really trying to process English text,
> you probably will want to write your own punctuation remover as it can
> then figure the difference between a period at the end of a sentence and
> a period at the end of abbreviations like "Dr." or "Mr."

Good point. A question mark does count as a word separator, so "token
1 of word 1 of..." will still work, but the other two could cause
problems.

-- 
-Mark Wieder
 mwieder at ahsoftware.net