parsing comments in scripts

[Alex Tweedly]

At 23:01 09/12/2004 -0800, Mark Wieder wrote:

>All-
>
>In the last few days I've started to realize how many ways comments
>and non-comments that look like comments can be embedded into lines of
>script code. In trying to get to what's actually code I've tried
>getting the offset of a "--" string, which has complications if it is
>quoted. So I tried looking for quoted text and seeing if the comment
>was after quoted text - this also presents problems. We can have:
>
>#comment
>text --comment
>text "more text -- still more"
>text "more text" --comment
>text "more text" text "-- text"
>text "more text" text "still more" --comment
>(did I leave any out)

text # comment
and variants of that.

>I came across an interesting combination of tokens and words: tokens
>ignore comments, words don't make that distinction. However, the token
>delimiter isn't necessarily where I want it to be:
>
>put the tokens of "put" && quote & "something" & quote && "--comment"
>results in
>put "something
>without the trailing quote.

That's not quite what I see - I get just
put something
i.e. without either quote; that makes sense because once the line is 
tokenized, the quotes are unnecessary.

>But using tokens as counters and words to get the text works. Bizarre
>but true (AFAIKT).

Sorry Mark, it won't work in all cases. In particular, any case where there 
are more tokens than words.

put a+b into c

has 6 tokens
   put
   a
   +
   b
   into
   c

but only 4 words
   put
   a+b
   into
   c

I don't have any suggestions, yet .... but it's an intriguing question, so 
I'll use it as an excuse to avoid gardening this afternoon :-)

-- Alex.