parsing comments in scripts

[Alex Tweedly]

At 13:01 10/12/2004 +0000, Alex Tweedly wrote:
>At 23:01 09/12/2004 -0800, Mark Wieder wrote:
>>I came across an interesting combination of tokens and words: tokens
>>ignore comments, words don't make that distinction. However, the token
>>delimiter isn't necessarily where I want it to be:
>>
>>put the tokens of "put" && quote & "something" & quote && "--comment"
>>results in
>>put "something
>>without the trailing quote.
>
>That's not quite what I see - I get just
>put something
>i.e. without either quote; that makes sense because once the line is 
>tokenized, the quotes are unnecessary.

Sorry - my error. I had already changed my script to investigate the case 
described later in my email (put a+b into c)

So (naturally enough, with hindsight) using something like
   token N
returns JUST the token itself, whereas
   token N to K
returns the intervening delimiters, but not any trailing one.

So we only need to restore any trailing quote, which is only needed if the 
final token started out being quoted, therefore

function StripComments theLine
   local theResult
   put token 1 to -1 of theLine into theResult
    if quote is in token -2 to -1 of theLine then put quote after theResult
   return theResult
end StripComents

should do it OK. Seems to work for my simple testing so far - but there may 
be cases I haven't thought of yet.

Unfortunately, it appears that token DOES include the C-style comment 
delimiters recently introduced, and they can be multi-line, so you'll need 
to do something more to deal with that.

-- Alex.