This introduces a (still-private) way to _push_ tokens into the parser,
rather than relying purely on a pull-based interface. Not only does this
simplify the iterator, but this is also preparing to make the new `feed_tok`
public so that parsers can be composed in more contexts. I suspect that
this method may also be useful for error recovery, since it can be used to
inject tokens into arbitrary points of a token stream.
I kept the new method private for now so that I can introduce the new API
and docs separate from this refactoring.
DEV-10863
The parsing framework originally created for XIR is now more general and
useful to other things. We'll see how this evolves.
This needs additional documentation, but I'd like to see how it changes as
I implement XmloReader and then some of the source readers first.
DEV-10863
Mike Gerwitz