A chunking mystery

[David Coker]

>  Consider (for starters) what happens if you have embedded commas in a
>  line like this:
>
>  "Los Angeles, CA",0,1
>
>  Column 3 in your CSV file is "1"
>  Item 3 is "0"
>
>  = Trouble!

You are certainly correct there, my friend!
In this case though, as part of the "process" for the raw source files
that are already in a somewhat usable CSV format, I first convert the
data into a piped "|" format, then strip out any extra quotes and
commas *and* shuffle the target field up to the 2nd item position.
Afterwards, I convert the data back to CSV format before doing any
additional processing. Ultimately, there are only three fields that I
am concerned with from a data processing viewpoint and everything else
in a given line just gets shuffled toward the right and ignored.

To the best of my ability, I've analyzed the resulting data and the
target fields are in the correct location, without any extra quotes
and/or commas. Since the data comes from several sources and I've had
similar results using a tab for the delimiter, Richmond may be on to
something in regards to what the Rev engine is actually "seeing".
Certainly must be considered and ruled out I think.

There again, as others have suggested, it may just be that some lines
are being skipped in certain cases due to the poor format of my repeat
loop.

Thanks for the ideas folks. Truly appreciated!
David