Word chunk includes punctuation

[Bob Sneidar]

Hmmm... odd again that 
put token one of word one of "test1!"

gives me
"test1" without the exclamation mark. 

The documentation states:
1. Each of the following characters is a token: =, +, -, *, /, [, ], (, ), {, }, <, >, and comma (,).

The exclamation mark is not one of those (unless I am token blind [again]) and yet it seems that it is being treated as one. 

This looks very much like an error in documentation. 

Bob


On Aug 13, 2012, at 1:32 PM, Paul Dupuis wrote:

> One caution: token does not separate . (period), ! (exclamation mark),
> or ? (question mark). If you are really trying to process English text,
> you probably will want to write your own punctuation remover as it can
> then figure the difference between a period at the end of a sentence and
> a period at the end of abbreviations like "Dr." or "Mr."
>