Tokenizer?

Mark Brownell gizmotron at earthlink.net
Fri Aug 13 09:21:43 CDT 2004


On Friday, August 13, 2004, at 06:39 AM, MisterX wrote:

> what is a tokenizer? ;0

If you are going to dream about features than dream big:

"When you are writing a lexical analyzer it would be helpful to have a 
class like the StreamTokenizer class from Sun's Java, so I've done 
something like that and here it is the CStringTokenizer class, the 
usage of this class is the same as by the StreamTokenizer from Java, 
there are a few additional functionality's and the function names are 
slightly different"
http://www.codeguru.com/Cpp/Cpp/cpp_mfc/parsing/article.php/c781/
----
Perhaps this could be part of getting to this:

Flex, version 2.5
A fast scanner generator

"Overview

This manual describes flex, a tool for generating programs that perform 
pattern-matching on text. The manual includes both tutorial and 
reference sections: "
http://www.gnu.org/software/flex/manual/html_mono/flex.html

Perhaps this could be the best approach to creating high speed specific 
need pull-parsers.

Mark




More information about the use-livecode mailing list