tokens and parsing

[Ivers, Doug E]

> -----Original Message-----
> From: Scott Raney [mailto:raney at metacard.com]
> Sent: Tuesday, January 08, 2002 2:16 PM
> To: use-revolution at lists.runrev.com
> Subject: Re: popups and tokens (was "Digest-something")
> 
> 
snip
> 
> > 2.
> > Seems that the word parser is little more than an item 
> parser with the
> > itemDelimiter set to " ",
> 
> It's a bit more, because it also skips multiple spaces, and tabs and
> returns in addition to spaces (neither is possible with item chunks).
> 
> > except for the stupid behavior with quotes.  I
> > would like a true word parser.  Or a parser for which we can specify
> > multiple delimiters.  Like a java token function.  What is 
> the best/fastest
> > way to parse words even in the presence of quotes and punctuation?
> 
> Use "token" chunks (e.g., "repeat for each token t in 
> <somecontainer>").
> It's the same parser the engine uses for compiling scripts.
> 
snip
> 

I did a little testing of the token and it doesn't seem to weed out chars
such as "." and "!".  So it appears that I will have to write my own word
parser.


-- D