I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least). And, as you can see, there's still a lot of TODO items here.
I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.
But, it works.
DEV-13708
Previous to this commit, ontological cross edges were declared
statically. But this doesn't fare well with the decided implementation for
template application.
The documentation details it, but we have Tpl->Ident which could mean "I
define this Ident once expanded", or it could mean "this is a reference to a
template I will be applying". The former is a tree edge, the latter is a
cross edge, and that determination can only be made by inspecting edge data
at runtime.
It could have been resolved by introducing new Object types, but that is a
lot of work for little benefit, especially given that only (right now) the
visitor uses this information.
DEV-13708
See the Air docblock for more information. I'm introducing new tokens for
the template system, which uses the terms "free" and "closed". I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.
DEV-13708
The intent was to have a very simple implementation of `hold_dangling` and
have everything work. But, I had a nasty surprise when the system tests
caught bug caused by some interesting depth interactions as it relates to
`xmli` and auto-closing.
I added an extra test/example in `asg::graph::visit::test` to illustrate the
situation; it was difficult to derive from the traces, but trivially obvious
once I wrote it out as an example.
With that, templates can now aggregate tokens for dangling expressions.
DEV-13708
This delegates expression parsing to `AirExprAggregate`, in an effort to
both begin to simplify the understanding and maintenance of `AirAggregate`;
and allow for parser composition for template parsing.
This utilizes the prior changes for token sum types to precisely define the
subset of AIR tokens supported by the expression parser. This differs from
prior approaches which delegated until a dead state, relying on runtime
information to determine if a parser has finished. This allows us to
determine that statically.
I do want to be able to eliminate the dead state from the parser so we can
get rid of the `unreachable!`, but I need to move on; that's something I had
tried to do in the past too, which ended up adding a bit of complexity, and
I'll have to consider my options in the future, including whether the dead
state transition can be entirely eliminated in favor of the combination of
these sum types and recovery; the parsing framework decisions were made
while recovery was still an open question, at least in practice.
DEV-13708
I wasn't initially sure whether I'd want separate tokens for different types
of identifying operations, but now that I see that it is clear from the
current state of the parser, there's no need.
This matches the name of the token in NIR.
DEV-13708
This is generic over the source, just as the target, defaulting just the
same to `ObjectIndex`.
This allows us to use only the edge information provided rather than having
to perform another lookup on the graph and then assert that we found the
correct edge. In this case, we're dealing with an `Ident->Expr` edge, of
which there is only one, but in other cases, there may be many such edges,
and it wouldn't be possible to know _which_ was referred to without also
keeping context of the previous edge in the walk.
So, in addition to avoiding more indirection and being more immune to logic
bugs, this also allows us to avoid states in `AsgTreeToXirf` for the purpose
of tracking previous edges in the current path. And it means that the tree
walk can seed further traversals in conjunction with it, if that is so
needed for deriving sources.
More cleanup will be needed, but this does well to set us up for moving
forward; I was too uncomfortable with having to do the separate
lookup. This is also a more intuitive API.
But it does have the awkward effect that now I don't need the pair---I just
need the `Object`---but I'm not going to remove it because I suspect I may
need it in the future. We'll see.
The TODO references the fact that I'm using a convenient `resolve_oi_pairs`
instead of resolving only the target first and then the source only in the
code path that needs it. I'll want to verify that Rust will properly
optimize to avoid the source resolution in branches that do not need it.
DEV-13708
This introduces a `Token` in place of the original tuple for
`TreePreOrderDfs` so that it can be used as input to a parser that will
lower into XIRF.
This requires that various things be describable (using `Display`), which
this also adds. This is an example of where the parsing framework itself
enforces system observability by ensuring that every part of the system can
describe its state.
DEV-13708
The `TreePreOrderDfs` iterator needed to expose additional edge context to
the caller (specifically, the `Span`). This was getting a bit messy, so
this consolodates everything into a new `DynObjectRel`, which also
emphasizes that it is in need of narrowing.
Packing everything up like that also allows us to return more information to
the caller without complicating the API, since the caller does not need to
be concerned with all of those values individually.
Depth is kept separate, since that is a property of the traversal and is not
stored on the graph. (Rather, it _is_ a property of the graph, but it's not
calculated until traversal. But, depth will also vary for a given node
because of cross edges, and so we cannot store any concrete depth on the
graph for a given node. Not even a canonical one, because once we start
doing inlining and common subexpression elimination, there will be shared
edges that are _not_ cross edges (the node is conceptually part of _both_
trees). Okay, enough of this rambling parenthetical.)
DEV-13708
This information is necessary to be able to reconstruct the tree, since
the `ObjectIndex` alone does not give you enough information. Even if you
inspected the graph, it _still_ wouldn't give you enough information, since
you don't know the current path of the traversal for nodes that may have
multiple incoming edges. (Any assumptions you could make today won't
always be valid in the future.)
DEV-13708
This begins to introduce a graph traversal useful for a source
reconstruction from the current state of the ASG. The idea is to, after
having parsed and ingested the source through the lowering pipeline, to
re-output it to (a) prove that we have parsed correctly and (b) allow
progressively moving things from the XSLT-based compiler into TAMER.
There's quite a bit of documentation here; see that for more
information. Generalizing this in an appropriate way took some time, but I
think this makes sense (that work began with the introduction of cross edges
in terms of the tree described by the graph's ontology). But I do need to
come up with an illustration to include in the documentation.
DEV-13708
Mike Gerwitz