I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
This creates another trait and struct `ObjectIndexToTree` that assert a
stronger invariant than `ObjectIndexRelTo`---that not only does it uphold
the invariants of `ObjectIndexRelTo`, but also that it represents a _tree_
edge, which indicates _ownership_ rather than just a reference.
This will be used to statically infer what can serve as a scope boundary for
upcoming changes. Specifically, anything that can own an `Ident` introduces
a new level of scope.
DEV-13708
This begins to introduce `AirStack` and starts to migrate context away from
the individual `ParseState`s onto the stack.
I should have started to commit earlier; this is getting a bit large and
makes it hard to follow what I'm doing so, hopefully stopping a little bit
short will allow the following commit to show that.
This is a work-in-progress change. All tests pass, but the refactoring is
incomplete. The `AirStack` abstraction is _also_ incomplete and will have
better, more domain-specific operations that make it harder to mess up
pairing pushes with pops.
The purpose of doing this is to allow `AirAggregate` to serve exclusively as
a sum state, which can then become a SuperState, much like `ele_parse!`'s
approach.
The _end_ goal of all of this is arbitrary template nesting.
DEV-13708
The graph's ontology is defined in the direction of the edge: from OA
to OB. This is enforced by the type system to ensure that no code path is
able to generate an invalid graph.
But that also makes it very difficult to work with a generic source to a
specific target.
This introduces a `ObjectIndexRelTo` trait that says whether `Self` is able
to be related to some `ObjectKind` `OB`, implements it for `ObjectIndex
where ObjectRelTo<OB>`, and introduces a new semi-opaque type
`ObjectIndexTo` that allows for the source `ObjectIndex` to be generic.
This then redefines some existing graph primitives in terms of
`ObjectIndexRelTo`, in particular creating edges, so that `ObjectIndex` can
be used as today, and the new `ObjectIndexTo` can be used in the same way
with the same API, without violating the graph ontology.
This will be used by `AirAggregate` to create dynamic targets for rooting
and splicing/expansion.
DEV-13708
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least). And, as you can see, there's still a lot of TODO items here.
I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.
But, it works.
DEV-13708
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.
Templates must not contribute to this environment until expansion. Further,
metavariables will not be present in this environment. To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed. This will also allow for the creation of more general
methods down the line.
DEV-13708
Previous to this commit, ontological cross edges were declared
statically. But this doesn't fare well with the decided implementation for
template application.
The documentation details it, but we have Tpl->Ident which could mean "I
define this Ident once expanded", or it could mean "this is a reference to a
template I will be applying". The former is a tree edge, the latter is a
cross edge, and that determination can only be made by inspecting edge data
at runtime.
It could have been resolved by introducing new Object types, but that is a
lot of work for little benefit, especially given that only (right now) the
visitor uses this information.
DEV-13708
This this a big change that's difficult to break up, and I don't have the
energy after it.
This introduces nullary template application, short- and long-form. Note
that a body of the short form is a `@values@` argument, so that's not
supported yet.
This continues to formalize the idea of what "template application" and
"template expansion" mean in TAMER. It makes a separate `TplApply`
unnecessary, because now application is simply a reference to a
template. Expansion and application are one and the same: when a template
expands, it'll re-bind metavariables to the parent context. So in a
template context, this amounts to application.
But applying a closed template will have nothing to bind, and so is
equivalent to expansion. And since `Meta` objects are not valid outside of
a `Tpl` context, applying a non-closed template outside of another template
will be invalid.
So we get all of this with a single primitive (getting the "value" of a
template).
The expansion is conceptually like `,@` in Lisp, where we're splicing trees.
It's a mess in some spots, but I want to get this committed before I do a
little bit of cleanup.
Also known as metavariables or template parameters.
This is a bit of a tortured excursion, trying to figure out how I want to
best represent this. I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.
`AirTpl::TplApply` highlights some remaining questions. What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such. But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue. But I'd rather address that head-on in
the future.
So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time. This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.
My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them. That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.
But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems
viable.
DEV-13708
This sets us up to be able to determine how `Dangling` expressions will be
rooted into templates.
This new strategy isn't yet handling `Dangling`; I wanted to get this
committed first so that the `Dangling` refactoring is more clear.
DEV-13708
This sets the stage for template parsing, and finally decides how we're
going to represent templates on the ASG. This is going to start simple,
since my original plans for improving how templates are
handled (conceptually) is going to have to wait.
This is the last difficult object type to figure out, with respect to graph
representation and derivation, so I wanted to get it out of the way.
DEV-13708
The previous commit demonstrated the amount of boilerplate necessary for
introducing new `ObjectKind`s; this abstracts away a lot of that
boilerplate, and allows for declarative relationship definition for the
ASG's ontology.
DEV-13708
There's quite a bit of boilerplate here that'll eventually need factoring
out. But it's also clear that it is somewhat onerous to add new object
types.
Note that a good chunk of this burden is _intentional_, via exhaustiveness
checks---adding a new type of object is an exceptional occurrence (well, in
principle, but we haven't added them all yet, so it'll be more common
initially), and we'd rather be safe to ensure that everything is properly
considering how that new type of object interacts with it.
Let's not confuse coupling with safety---the latter causes a burden because
of the former, not because of itself; it provides a service to us.
But, nonetheless, we'll want to reduce this burden somewhat since there are
a number more to add.
DEV-13708
Mike Gerwitz