Of course I would run into integration issues. My foresight is lacking.
The purpose of this is to allow for type narrowing before passing data to a
more specialized ParseState, so that the other ParseState doesn't need to
concern itself with the entire domain of inputs that it doesn't need, and
repeat unnecessary narrowing.
For example, consider XIRF: it has an `Attr` variant, which holds an `Attr`
object. We'll want to desugar that object. It does not make sense to
require that the desugaring process accept `XirfToken` when we've already
narrowed it to an `Attr`---we should accept an Attr.
However, we run into a problem immediately: what happens with tokens that
bubble back up due to lookahead or errors? Those tokens need to be
converted _back_ (widened). Fortunately, widening is a much easier process
than narrowing---we can simply use `From`, as we do today so many other
places.
So, this still keeps the onus of narrowing on the caller, but for now that
seems most appropriate. I suspect Rust would optimize away duplicate
checks, but that still leaves the maintenance concern---the two narrowings
could get out of sync, and that's not acceptable.
Unfortunately, this is just one of the problems with integration...
DEV-13156
My initial plan with expansion was to wrap a `PasteState` in another that
unwraps `Expansion` and converts into a `Dead` state, so that existing
`TransitionResult` stitching methods (`delegate`, specifically) could be
used.
But the desire to use that existing method was primarily because stitching
was a complex operation that was abstracted away _as part of the `delegate`
method_, which made writing new ones verbose and difficult. Thus began the
previous commits to begin to move that responsibility elsewhere so that it
could be more composable.
This continues with that, introducing a new trait that will culminate in the
removal of a wrapping `ParseState` in favor of a stitching method. The old
`StitchableExpansionState` is still used for tests, which demonstrates that
the boilerplate problem still exists despite improvements made here These
will become more generalized in the future as I have time (and the
functional aspects of the code more formalized too, now that they're taking
shape).
The benefit of this is that we avoid having to warp our abstractions in ways
that don't make sense (use of a dead state transition) just to satisfy
existing APIs. It also means that we do not need the boilerplate of a
`ParseState` any time we want to introduce this type of
stitching/delegation. It also means that those methods can eventually be
extracted into more general traits in the future as well.
Ultimately, though, the two would have accomplished the same thing. But the
difference is most emphasized in the _parent_---the actual stitching still
has to take place for desugaring in the attribute parser, and I'd like for
that abstraction to still be in terms of expansion. But if I utilized
`StitchableExpansionState`, which converted into a dead state, I'd have to
either forego the expansion abstraction---which would make the parser even
more confusing---or I'd have to create _another_ abstraction around the dead
state, which would mean that I stripped one abstraction just to introduce
another one that's essentially the same thing. It didn't feel right, but it
would have worked.
The use of `PhantomData` in `StitchableExpansionState` was also a sign that
something wasn't quite right, in terms of how the abstractions were
integrating with one-another.
And so here we are, as I struggle to wade my way through all of the yak
shavings and make any meaningful progress on this project, while others
continue to suffer due to slow build times.
I'm sorry. Even if the system is improving.
DEV-13156
There's no use in duplicating this in util::expand.
Lookahead tokens are one of the few invariants that I haven't taken the time
of enforcing using the type system, because it'd be quite a bit of work that
I do not have time for, and may not be worth it with changes that may make
the system less ergonomic. Nonetheless, I do hope to address it at some
point in the (possibly-far) future.
If ever you encounter this diagnostic message, ask yourself how stable TAMER
otherwise is and how many other issues like this have been entirely
prevented through compile-time proofs using the type system.
DEV-13156
This has evolved into a more robust and independent concept, but it is still
a utility in the sense that it's utilizing existing parsing framework
features and making them more convenient.
DEV-13156
These traits serve to abstract away some of the type-level details and
clearly state what the end result is (something stitchable with a parent).
I'm admittedly battling myself on this concept a bit. The proper layer of
abstraction is the concept of expansion, which is an abstraction that is
likely to be maintained all the way through, but we strip the abstraction
for the sake of delegation. Maybe the better option is to provide a
different method of delegation and avoid the stripping at all, and avoid the
awkward interaction with the dead state.
The awkwardness comes from the fact that delegating right now is so rigid
and defined in terms of a method on state rather than a mapping between
`TransitionResult`s. But I really need to move on... ;_;
The original design was trying to generalize this such that composition at
the attribute parser level (for NIR) would be able to just accept any
sitchable parser with the convention that the dead state is the replacement
token. But that is the wrong layer of abstraction, which not only makes it
confusing, but is asking for trouble when someone inevitably violates that
contract.
With all of that said, `StitchableExpansionState` _is_ a delegation. It
could just as easily be a function (`is_accepting` always delegates too), so
perhaps that should just be generalized as reifying delegation as a
`ParseState`.
DEV-13156
This parser really just allows me to continue developing the NIR
interpolation system using `Expansion` terminology, and avoid having to use
dead states in tests. This allows for the appropriate level of abstraction
to be used in isolation, and then only be stripped when stitching is
necessary.
Future commits will show how this is actually integrated and may introduce
additional abstraction to help.
DEV-13156
Mike Gerwitz