#ifndef CPPTOKER_H // compile if CPPTOKER_H is not in symbol table
#define CPPTOKER_H // put CPPTOKER_H in symbol table
/////////////////////////////////////////////////////////////////////////
// CppToker.h - declares tokenizer which extracts words from a stream //
// ver 1.1 //
// Language: Visual C++, ver 7.0 //
// Platform: Dell Dimension 8300, Windows XP Prof, SP 1 //
// Application: CSE687 prototype for project #1, Spring 2004 //
// Author: Jim Fawcett, CST 2-187, Syracuse University //
Function Declaration Start
// (315) 443-3948, jfawcett@twcny.rr.com //
/////////////////////////////////////////////////////////////////////////
/*
* Module Operations:
* ==================
Class Declaration
* CppToker.h Declares a class CToker designed to extract words from a
* std stream. Each word is either an identifier or punctuator token.
* Identifiers consist of alphabetic characters, digits, and underscores.
* Punctuators are composed of all other non-whitespace characters. A
* specific set of punctuation characters are designated as single character
* tokens. All other non-identifiers consist of multiple punctuators.
*
* Public Interface:
* =================
* std::fstream in(someFile);
* CToker toker(&in); // create toker
* std::string tok;
while((tok = toker.getTok()) != "") // read tokens
* // do something with tok
* std::istream *pStr = toker.getStream(); // get pointer to stream
* std::istream *pOld = toker.setStream(pNew); // give toker new stream
* std::string curr = toker.curr(); // get last token extracted
* std::string prev = toker.prev(); // get one before that
*/
/*
* Maintenance History:
* ====================
* ver 1.1 : 08 Feb 04
* - added '~' to set of single character tokens in CTokBuffer constructor
* ver 1.0 : 12 Jan 04
* - first release
*
* Build Process:
* ==============
* Required Files:
* CppToker.h, CppToker.cpp
*
* Compiler Command:
* cl /DTest_CPPTOKER /EHsc CppToker.cpp;
*/
//
/*
* Design Notes:
* =============
* This module illustrates many things we will be covering this semester, some
* rather unusual things, for purposes of illustration:
* - the type std::basic_string<char> appears in this header file. You will
* usually see the type std::string in most C++ code, which is equivalent.
* This syntax makes it clear that the standard string is a template-based
* type, using the 8 bit Char type as part of its definition. We will be
* saying much more about this later in the course.
Class Declaration
* - The CToker class has a private helper class, CTokBuffer, and declares a
Class Declaration
* reference to that type. This prevents CToker class from participating
* in copy or assignment operations. That is consistent with my intended
Class Declaration
* abstraction for this class.
Class Declaration
* - I wanted to keep CTokBuffer out of the abstraction of the CToker class
* For that reason its declaration is in the implementation file,
* CppToker.cpp, instead of in this header file. A consequence of that is
Class Declaration
* that the CToker class cannot hold an object of CTokBuffer, because its
* size is not known to the compiler yet. It could hold either a pointer
* or a reference to its CTokBuffer instance. I chose to use a reference
* so that I could use object semantics rather than pointer semantics in
* the CToker member function implementations.
*/
#include <iostream>
#include <vector>
#include <string>
#include <set>
namespace Scanner
{
///////////////////////////////////////////////////////////////////////
Class Declaration
// class CToker - extracts tokens from a std stream
Class Declaration
// - uses a private CTokBuffer class to help build
// tokens from stream
Class Declaration
class CToker
{
public:
CToker(std::basic_istream<char> *pStream=0);
~CToker(void);
std::istream* setStream(std::basic_istream<char> *pStream);
std::basic_string<char> getTok();
std::istream* getStream();
std::basic_string<char> prev();
std::basic_string<char> curr();
private:
std::istream *_pStream;
Class Declaration
// forward reference to helper class
Class Declaration
class CTokBuffer;
// reference type below will prevent CToker objects
// being copied or assigned
CTokBuffer &tokBuffer;
std::basic_string<char> prevTok;
std::basic_string<char> currTok;
};
End of Class Declaration
}
#endif