tokenizer contains a single class, Toker, responsible for extracting tokens from a stream. Since it refers
to the istream class, it can also work with streams derived from istream, like istringstream.
Elaboration:
"
If a class has more than one responsibility, then the responsibilities become coupled.
Changes to one responsibility may impare or inhibit the ability of the class to meet
the others. This kind of coupling leads to fragile designs that break in unexpected
ways when changed.
"
- Robert C. Martin, Agile Software Development, Prentice-Hall