Bug with token chunk type?

[Dave Cragg]

On 20 Mar 2007, at 05:47, Richard Gaskin wrote:
> Why should "s" be parsed as a separate "token" from the numeric  
> portion  if it trails or is anywhere in the middle, but only if  
> it's at the beginning then it's considered a part of the  
> alphanumeric string? :\
>
> Unless someone can come up with a good reason why this is what one  
> should expect I'll log it as a bug tomorrow.

Does there need to be a good reason? From the docs, "The token chunk  
is jmplemented for the Transcript language, and probably isn't  
suitable for use in a general-purpose language parser."

I've always taken this to mean that "token" is the engine's private  
property, and we have no basis for questionning how it works. We can  
guess, but we can also expect to be surprised.

Cheers
Dave