Word chunk includes punctuation

[Paul Dupuis]

Peter,

That would be great. Even better would be to post as a user comment to
the Live Code Dictionary so the correct information is available to
everyone who look up "token" in the Dictionary.

Or if you prefer to post here, would you mind if I or someone then added
it to the Dictionary entry?

On 8/13/2012 8:14 PM, Peter Haworth wrote:
> I've used tokens quite a bit for parsing sqlite statements and the docs are
> definitely incomplete.  I'll try to rebuild what I learned and publish it
> to the list.
> Pete
> lcSQL Software <http://www.lcsql.com>
>
>
>
> On Mon, Aug 13, 2012 at 2:01 PM, Bob Sneidar <bobs at twft.com> wrote:
>
>> Hmmm... odd again that
>> put token one of word one of "test1!"
>>
>> gives me
>> "test1" without the exclamation mark.
>>
>> The documentation states:
>> 1. Each of the following characters is a token: =, +, -, *, /, [, ], (, ),
>> {, }, <, >, and comma (,).
>>
>> The exclamation mark is not one of those (unless I am token blind [again])
>> and yet it seems that it is being treated as one.
>>
>> This looks very much like an error in documentation.
>>
>> Bob
>>
>>
>> On Aug 13, 2012, at 1:32 PM, Paul Dupuis wrote:
>>
>>> One caution: token does not separate . (period), ! (exclamation mark),
>>> or ? (question mark). If you are really trying to process English text,
>>> you probably will want to write your own punctuation remover as it can
>>> then figure the difference between a period at the end of a sentence and
>>> a period at the end of abbreviations like "Dr." or "Mr."
>>>
>>
>> _______________________________________________
>> use-livecode mailing list
>> use-livecode at lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>

-- 
Paul Dupuis
Cofounder
Researchware, Inc.
http://www.researchware.com/
http://www.twitter.com/researchware
http://www.facebook.com/researchware
http://www.linkedin.com/company/researchware-inc