parsing comments in scripts

[Mark Wieder]

All-

In the last few days I've started to realize how many ways comments
and non-comments that look like comments can be embedded into lines of
script code. In trying to get to what's actually code I've tried
getting the offset of a "--" string, which has complications if it is
quoted. So I tried looking for quoted text and seeing if the comment
was after quoted text - this also presents problems. We can have:

#comment
text --comment
text "more text -- still more"
text "more text" --comment
text "more text" text "-- text"
text "more text" text "still more" --comment
(did I leave any out)

I came across an interesting combination of tokens and words: tokens
ignore comments, words don't make that distinction. However, the token
delimiter isn't necessarily where I want it to be:

put the tokens of "put" && quote & "something" & quote && "--comment"
results in
put "something
without the trailing quote.
But using tokens as counters and words to get the text works. Bizarre
but true (AFAIKT).

So... herewith I present my parser for removing comments from lines of
code in a script. I believe it does everything except for line
continuation characters ("\") which I handle separately before calling
this function.

---------------------
-- Return an empty line if it starts with "#"
-- If the line contains an embedded comment ("--") then
-- delete the comment part and just return the Transcript content
---------------------
function StripComments theLine
  local numTokens
  
  -- ignore comments
  put the number of tokens of theLine into numTokens
  if numTokens is zero then
    put empty into theLine
  else
    put word 1 to numTokens of theLine into theLine
  end if
  return theLine
end StripComments

-- 
-Mark Wieder
 mwieder at ahsoftware.net