Tokenizer?

[Mark Brownell]

On Friday, August 13, 2004, at 06:39 AM, MisterX wrote:

> what is a tokenizer? ;0

If you are going to dream about features than dream big:

"When you are writing a lexical analyzer it would be helpful to have a 
class like the StreamTokenizer class from Sun's Java, so I've done 
something like that and here it is the CStringTokenizer class, the 
usage of this class is the same as by the StreamTokenizer from Java, 
there are a few additional functionality's and the function names are 
slightly different"
http://www.codeguru.com/Cpp/Cpp/cpp_mfc/parsing/article.php/c781/
----
Perhaps this could be part of getting to this:

Flex, version 2.5
A fast scanner generator

"Overview

This manual describes flex, a tool for generating programs that perform 
pattern-matching on text. The manual includes both tutorial and 
reference sections: "
http://www.gnu.org/software/flex/manual/html_mono/flex.html

Perhaps this could be the best approach to creating high speed specific 
need pull-parsers.

Mark