Thinking In C++. Volume 2: Practical Programming

The following version shows how to build the list of words with an istreambuf_iterator that moves the characters from one place (the input stream) to another (a string object), depending on whether the Standard C library function isalpha( ) returns true:

//: C07:WordList2.cpp

// Illustrates istreambuf_iterator and insert iterators

#include

#include

#include

#include

#include

#include

#include "../require.h"

using namespace std;

int main(int argc, char* argv[]) {

  char* fname = "WordList2.cpp";

  if(argc > 1) fname = argv[1];

  ifstream in(fname);

  assure(in, fname);

  istreambuf_iterator p(in), end;

  set wordlist;

  while (p != end) {

    string word;

    insert_iterator

      ii(word, word.begin());

    // Find the first alpha character:

    while(!isalpha(*p) && p != end)

      p++;

    // Copy until the first non-alpha character:

    while (isalpha(*p) && p != end)

      *ii++ = *p++;

    if (word.size() != 0)

      wordlist.insert(word);

  }

  // Output results:

  copy(wordlist.begin(), wordlist.end(),

    ostream_iterator(cout, "\n"));

} ///:~

This example was suggested by Nathan Myers, who invented the istreambuf_iterator and its relatives. This iterator extracts information character by character from a stream. Although the istreambuf_iterator template argument might suggest that you could extract, for example, ints instead of char, that’s not the case. The argument must be of some character type—a regular char or a wide character.

After the file is open, an istreambuf_iterator called p is attached to the istream so characters can be extracted from it. The set called wordlist will hold the resulting words.

The while loop reads words until the end of the input stream is found. This is detected using the default constructor for istreambuf_iterator, which produces the past-the-end iterator object end. Thus, if you want to test to make sure you’re not at the end of the stream, you simply say p != end.

The second type of iterator that’s used here is the insert_iterator, which creates an iterator that knows how to insert objects into a container. Here, the "container" is the string called word, which, for the purposes of insert_iterator, behaves like a container. The constructor for insert_iterator requires the container and an iterator indicating where it should start inserting the characters. You could also use a back_insert_iterator, which requires that the container have a push_back( ) (string does).

After the while loop sets everything up, it begins by looking for the first alpha character, incrementing start until that character is found. It then copies characters from one iterator to the other, stopping when a nonalpha character is found. Each word, assuming it is nonempty, is added to wordlist.

A completely reusable tokenizer

The word list examples use different approaches to extract tokens from a stream, neither of which is very flexible. Since the STL containers and algorithms all revolve around iterators, the most flexible solution will itself use an iterator. You could think of the TokenIterator as an iterator that wraps itself around any other iterator that can produce characters. Because it is certainly a type of input iterator (the most primitive type of iterator), it can provide input to any STL algorithm. Not only is it a useful tool in itself, the following TokenIterator is also a good example of how you can design your own iterators.[97]

The TokenIterator class is doubly flexible. First, you can choose the type of iterator that will produce the char input. Second, instead of just saying what characters represent the delimiters, TokenIterator will use a predicate that is a function object whose operator( ) takes a char and decides whether it should be in the token. Although the two examples given here have a static concept of what characters belong in a token, you could easily design your own function object to change its state as the characters are read, producing a more sophisticated parser.

The following header file contains two basic predicates, Isalpha and Delimiters, along with the template for TokenIterator:

//: C07:TokenIterator.h

#ifndef TOKENITERATOR_H

#define TOKENITERATOR_H

#include

#include

#include

#include

#include

struct Isalpha : std::unary_function {

  bool operator()(char c) {

    return std::isalpha(c);

  }

};

class Delimiters : std::unary_function {

  std::string exclude;

public:

  Delimiters() {}

  Delimiters(const std::string& excl)

    : exclude(excl) {}

  bool operator()(char c) {

    return exclude.find(c) == std::string::npos;

  }

};

template

class TokenIterator : public std::iterator<

    std::input_iterator_tag,std::string, std::ptrdiff_t> {

  InputIter first;

  InputIter last;

  std::string word;

  Pred predicate;

public:

  TokenIterator(InputIter begin, InputIter end,