Thinking In C++. Volume 2: Practical Programming

    Pred pred = Pred())

    : first(begin), last(end), predicate(pred) {

      ++*this;

  }

  TokenIterator() {} // End sentinel

  // Prefix increment:

  TokenIterator& operator++() {

    word.resize(0);

    first = std::find_if(first, last, predicate);

    while (first != last && predicate(*first))

      word += *first++;

    return *this;

  }

  // Postfix increment

  class Proxy {

    std::string word;

  public:

    Proxy(const std::string& w) : word(w) {}

    std::string operator*() { return word; }

  };

  Proxy operator++(int) {

    Proxy d(word);

    ++*this;

    return d;

  }

  // Produce the actual value:

  std::string operator*() const { return word; }

  std::string* operator->() const {

    return &(operator*());

  }

  // Compare iterators:

  bool operator==(const TokenIterator&) {

    return word.size() == 0 && first == last;

  }

  bool operator!=(const TokenIterator& rv) {

    return !(*this == rv);

  }

};

#endif // TOKENITERATOR_H ///:~

The TokenIterator class derives from the std::iterator template. It might appear that some kind of functionality comes with std::iterator, but it is purely a way of tagging an iterator so that a container that uses it knows what it’s capable of. Here, you can see input_iterator_tag as the iterator_category template argument—this tells anyone who asks that a TokenIterator only has the capabilities of an input iterator and cannot be used with algorithms requiring more sophisticated iterators. Apart from the tagging, std::iterator doesn’t do anything beyond providing several useful type definitions, which means you must design all the other functionality in yourself.

The TokenIterator class may look a little strange at first, because the first constructor requires both a "begin" and an "end" iterator as arguments, along with the predicate. Remember, this is a "wrapper" iterator that has no idea how to tell whether it’s at the end of its input source, so the ending iterator is necessary in the first constructor. The reason for the second (default) constructor is that the STL algorithms (and any algorithms you write) need a TokenIterator sentinel to be the past-the-end value. Since all the information necessary to see if the TokenIterator has reached the end of its input is collected in the first constructor, this second constructor creates a TokenIterator that is merely used as a placeholder in algorithms.

The core of the behavior happens in operator++. This erases the current value of word using string::resize( ) and then finds the first character that satisfies the predicate (thus discovering the beginning of the new token) using find_if( ) (from the STL algorithms, discussed in the following chapter). The resulting iterator is assigned to first, thus moving first forward to the beginning of the token. Then, as long as the end of the input is not reached and the predicate is satisfied, input characters are copied into word. Finally, the TokenIterator object is returned and must be dereferenced to access the new token.

The postfix increment requires a proxy object to hold the value before the increment, so it can be returned. Producing the actual value is a straightforward operator*. The only other functions that must be defined for an output iterator are the operator== and operator!= to indicate whether the TokenIterator has reached the end of its input. You can see that the argument for operator== is ignored—it only cares about whether it has reached its internal last iterator. Notice that operator!= is defined in terms of operator==.

A good test of TokenIterator includes a number of different sources of input characters, including a streambuf_iterator, a char*, and a deque::iterator. Finally, the original word list problem is solved:

//: C07:TokenIteratorTest.cpp

//{-msc}

#include "TokenIterator.h"

#include "../require.h"

#include

#include

#include

#include

#include

using namespace std;

int main(int argc, char* argv[]) {

  char* fname = "TokenIteratorTest.cpp";

  if(argc > 1) fname = argv[1];

  ifstream in(fname);

  assure(in, fname);

  ostream_iterator out(cout, "\n");

  typedef istreambuf_iterator IsbIt;

  IsbIt begin(in), isbEnd;

  Delimiters

    delimiters(" \t\n~;()\"<>:{}[]+-=&*#.,/\\");

  TokenIterator

    wordIter(begin, isbEnd, delimiters),

    end;

  vector wordlist;

  copy(wordIter, end, back_inserter(wordlist));

  // Output results:

  copy(wordlist.begin(), wordlist.end(), out);

  *out++ = "-----------------------------------";

  // Use a char array as the source:

  char* cp =

    "typedef std::istreambuf_iterator It";

  TokenIterator

    charIter(cp, cp + strlen(cp), delimiters),

    end2;

  vector wordlist2;

  copy(charIter, end2, back_inserter(wordlist2));

  copy(wordlist2.begin(), wordlist2.end(), out);

  *out++ = "-----------------------------------";

  // Use a deque as the source:

  ifstream in2("TokenIteratorTest.cpp");

  deque dc;