I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
All ObjectIndex-like objects hash using only the underlying identifier,
which ultimately boils down to a `NodeIndex` (petgraph), which is just a
u32. And so in that sense, the only purpose we have for hashing it is to
(a) reduce the space required to store mappings, and (b) compose with other
`Hash`es.
DEV-13708
This creates another trait and struct `ObjectIndexToTree` that assert a
stronger invariant than `ObjectIndexRelTo`---that not only does it uphold
the invariants of `ObjectIndexRelTo`, but also that it represents a _tree_
edge, which indicates _ownership_ rather than just a reference.
This will be used to statically infer what can serve as a scope boundary for
upcoming changes. Specifically, anything that can own an `Ident` introduces
a new level of scope.
DEV-13708
This allows this method to be used on anything that is able to relate to an
identifier, which is needed for the changes being made for the template
system.
This linear lookup is actually going away (as hinted at by preceding
commits); this is extracted as part of a larger change and I wanted to get
it committed to make it easier to follow upcoming changes.
DEV-13708
The prior commit begins to explain the end goal of being able to index
identifiers outside of the global environment.
This change continues to index things as before, but introduces a new key
based on the pair of the symbol id together with a node that is _part of_
its target environment. The only environment utilized at the moment (in this
commit) is that of the root node (which is the global scope), in both
indexing and lookup. Future commits will extend this, and contain more
information about and rationale for the implementation.
The new general index methods are restricted to `pub(super)` until an
abstraction can be put in place that is responsible for environment
indexing; that's a responsibility that is currently handled by
`AirAggregateCtx` for tamec, and the linker has no scoping
requirements since all of that has already been dealt with.
DEV-13708
This reverts commit 1b7eac337cd5909c01ede3a5b3fba577898d5961.
This is a revert of the previous revert, just so that I (and you) have
references to prior rationale.
This was previously reverted because it wasn't worth doing, but now we have
a situation where we need to begin implementing lexical scoping rules for
nested containers (packages and templates). In particular, as you'll see in
the commits that follow, we need to be able to look up an identifier that
may have been created as Missing at one level of scope (certain types of
blocks), but then define it at another level.
Or, even more simply at this point, since I'm not yet doing anything
sophisticated with scope: we're only indexing in the global environment, and
we need to be able to index elsewhere too.
The next commit will go into more information, but suffice it to say for now
that indexing is going to get more complicated than a SymbolId.
Sticking with FxHash for now; we don't need a stable hash now.
DEV-13708
There are no such invalid expansion contexts yet, but this gets rid of the
final remaining TODO from introducing the stack. With the existing feature
set, at least.
DEV-13708
This eliminates the TODOs that existed when looking for an OI for rooting an
identifier.
The change to `rooting_ci` is ridiculous, but I want to get other things
done before I jump down the rabbit hole of generalizing that (indexing local
identifiers). Though I have an approach in mind.
DEV-13708
Just some continued cleanup.
Unfortunately, we have sacrificed knowing a package OI must exist
statically, even though one will always be available.
DEV-13708
This has AirAggregate preempt Expr parsing in the same way as templates,
rather than having `AirTplAggregate` concern itself with expression
tokens. This continues to simplify `AirTplAggregate`, which was getting
quite complex not too long ago.
A pattern is now emerging for the call/ret convention for preemption. That
was intentional, but it's nice to see it manifest so obviously before I
abstract it away.
DEV-13708
This was extracted from xir::parse::ele in previous commits. The
conventions help to ensure that pushing and returns are being performed
correctly. The abstraction will continue to evolve.
This ends up using `Ready` as the dead state. I need to determine if this
is ideal, and if so, maybe just use `Default`, otherwise yield an error.
DEV-13708
`AirAggregate` now handles all delegation to `AirExprAggregate`. This is
possible because `AirAggregate` is now the superstate for each of these
parsers, so `AirTplAggregate` is able to transition to a state that is not
its own.
This does not go so far as reaching the ultimate objective---having nested
template support---even though it'd be fairly simple to do now; there's
going to be a number of interesting consequences to these changes, and a bit
of cleanup is still needed, and I want tests observing this functionality to
accompany those changes. That is: let's keep this a refactoring, to the
extent that it's possible.
Things are getting much easier to understand now, and much cleaner.
DEV-13708
I love deleting code I just wrote...
This doesn't solve the underlying problems with identifiers, but it does at
least lift it into the `AirAggregateCtx`, allowing `AirExprAggregate` to be
even further simplified. Now the `From` implementation is not specialized
and we can readily convert to a SuperState.
There's still a lot of TODOs here, though. And some of them will
unfortunately require runtime checks where there was previously a
compile-time check. But that's okay in a lot of the cases, because the
empty behavior will replace existing error checks.
DEV-13708
Whether or not dangling expressions are permitted is now based solely off of
the stack context, which is also much more intuitive.
`RootStrategy` now only does one thing, and the existing comments describe
why it exists despite that one thing seeming very similar.
`RootStrategy` further alludes to how `ExprStack` could also be
eliminated, should it be worth doing so. It is a tad redundant now with the
new stack.
DEV-13708
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
This is more of the same of the previous commit, but in a more digestable
chunk. We now have child states that are able to be constructed using a
simple `From`, which is important to making `AirAggregate` a `SuperState`.
This also makes `AirStack` act like a prototype chain for `ObjectIndex`es,
creating environments where context shadows. The linear search should only
have to check the last two frames (e.g. an Expr has a parent Pkg or Tpl
context which will have a `rooting_oi` value), and this is only done during
a rooting operation.
DEV-13708
This begins to introduce `AirStack` and starts to migrate context away from
the individual `ParseState`s onto the stack.
I should have started to commit earlier; this is getting a bit large and
makes it hard to follow what I'm doing so, hopefully stopping a little bit
short will allow the following commit to show that.
This is a work-in-progress change. All tests pass, but the refactoring is
incomplete. The `AirStack` abstraction is _also_ incomplete and will have
better, more domain-specific operations that make it harder to mess up
pairing pushes with pops.
The purpose of doing this is to allow `AirAggregate` to serve exclusively as
a sum state, which can then become a SuperState, much like `ele_parse!`'s
approach.
The _end_ goal of all of this is arbitrary template nesting.
DEV-13708
The graph's ontology is defined in the direction of the edge: from OA
to OB. This is enforced by the type system to ensure that no code path is
able to generate an invalid graph.
But that also makes it very difficult to work with a generic source to a
specific target.
This introduces a `ObjectIndexRelTo` trait that says whether `Self` is able
to be related to some `ObjectKind` `OB`, implements it for `ObjectIndex
where ObjectRelTo<OB>`, and introduces a new semi-opaque type
`ObjectIndexTo` that allows for the source `ObjectIndex` to be generic.
This then redefines some existing graph primitives in terms of
`ObjectIndexRelTo`, in particular creating edges, so that `ObjectIndex` can
be used as today, and the new `ObjectIndexTo` can be used in the same way
with the same API, without violating the graph ontology.
This will be used by `AirAggregate` to create dynamic targets for rooting
and splicing/expansion.
DEV-13708
To simplify things in support of upcoming changes, we'll just instantiate a
new one as needed. This doesn't have an appreciable performance impact, so
the optimization is premature. It was done just because it was more of the
same that TAMER was already doing, but now it's making things more
difficult.
DEV-13708
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
Previously, `AirTplAggregate` worked only in a `Pkg` context, being able to
root `Tpl` `Ident`s in `Pkg` and expand only into `Pkg`. This still does
the same, but generalizes to allow for different roots and expansion
targets.
This will be utilized to parse nested templates.
DEV-13708
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least). And, as you can see, there's still a lot of TODO items here.
I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.
But, it works.
DEV-13708
I'm not happy with this implementation. The linear search is undesirable,
but not too bad (and maybe wouldn't even be worth caching, if this were the
whole story), but we _also_ need to prevent duplicate identifiers. We are
not going to want to perform a linear search of a linked list (effectively)
every time we add an identifier to check for uniqueness, so I think the
caching is going to have to be generalized very shortly anyway.
As it stands now, a duplicate identifier would cause an error at expansion
time. That's not what we want, but it's not terrible, because you can have
that same problem in normal circumstances without local conflicts.
But this'll be used for metavariables as well, where we absolutely _do_ want
to fail at template definition time.
DEV-13708
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.
Templates must not contribute to this environment until expansion. Further,
metavariables will not be present in this environment. To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed. This will also allow for the creation of more general
methods down the line.
DEV-13708
This makes the tests quite a bit easier to understand visually. I've been
doing this with all new tests but had to go back to some old ones, and still
have more to go back to. Baby steps.
DEV-13708
Previous to this commit, ontological cross edges were declared
statically. But this doesn't fare well with the decided implementation for
template application.
The documentation details it, but we have Tpl->Ident which could mean "I
define this Ident once expanded", or it could mean "this is a reference to a
template I will be applying". The former is a tree edge, the latter is a
cross edge, and that determination can only be made by inspecting edge data
at runtime.
It could have been resolved by introducing new Object types, but that is a
lot of work for little benefit, especially given that only (right now) the
visitor uses this information.
DEV-13708
This this a big change that's difficult to break up, and I don't have the
energy after it.
This introduces nullary template application, short- and long-form. Note
that a body of the short form is a `@values@` argument, so that's not
supported yet.
This continues to formalize the idea of what "template application" and
"template expansion" mean in TAMER. It makes a separate `TplApply`
unnecessary, because now application is simply a reference to a
template. Expansion and application are one and the same: when a template
expands, it'll re-bind metavariables to the parent context. So in a
template context, this amounts to application.
But applying a closed template will have nothing to bind, and so is
equivalent to expansion. And since `Meta` objects are not valid outside of
a `Tpl` context, applying a non-closed template outside of another template
will be invalid.
So we get all of this with a single primitive (getting the "value" of a
template).
The expansion is conceptually like `,@` in Lisp, where we're splicing trees.
It's a mess in some spots, but I want to get this committed before I do a
little bit of cleanup.
This chooses Option B, as stated would likely be the case in the previous
commit. The reasons are practical---I intend to support partial application
if doing so is worth it, either in implementation of the compiler or the
source language.
Closed templates can be referenced using `IdentRef` to trigger
expansion---their value is what they expand into, and they are spliced into
that point in the tree, like `,@` in Lisp. We are able to overload this
behavior because we have the necessary type information.
However, I don't want to have to generate an Ident for every single template
expansion; there are many tens of thousands of them in our production
system. Since AIR doesn't presently have a way to deal with this situation,
I'll for now add a special token that will close and expand a template in
place; it can be replaced with two separate tokens (`TplEnd` + `Ref`, for
example) in the future if such a need arises.
Are we there yet...?
DEV-13708
Also known as metavariables or template parameters.
This is a bit of a tortured excursion, trying to figure out how I want to
best represent this. I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.
`AirTpl::TplApply` highlights some remaining questions. What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such. But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue. But I'd rather address that head-on in
the future.
So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time. This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.
My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them. That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.
But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems
viable.
DEV-13708
See the Air docblock for more information. I'm introducing new tokens for
the template system, which uses the terms "free" and "closed". I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.
DEV-13708
This removes special cases, but it does complicate the parent `AirAggregate`
parser. A pattern of delegation is forming, though abstracting it may be an
interesting challenge, given Rust's limitation on macro invocations as match
arms. But, I think I can manage by generating the entire match using a
macro with a match-compatible syntax, augmenting where
needed...maybe. This'll be messy.
...but if I can write the nightmare that is `ele_parse!`, I'm sure I can
manage this. I just prefer to avoid complex macros unless I really need
them.
DEV-13708
Small break from templates for something easier. I have COVID-19, so I'll
use that as my excuse for wanting to be more lazy.
The real reason is to see some more concrete progress and ensure that
patterns hold for simple expressions before further refactoring.
But, before I proceed with such refactoring, I really ought to approach
something that requires a NIR desugaring step, like case statements.
DEV-13708
The intent was to have a very simple implementation of `hold_dangling` and
have everything work. But, I had a nasty surprise when the system tests
caught bug caused by some interesting depth interactions as it relates to
`xmli` and auto-closing.
I added an extra test/example in `asg::graph::visit::test` to illustrate the
situation; it was difficult to derive from the traces, but trivially obvious
once I wrote it out as an example.
With that, templates can now aggregate tokens for dangling expressions.
DEV-13708
And finally we have tokens aggregated onto the ASG in the context of a
template. I expected to arrive here much more quickly, but there was a lot
of necessary refactoring. There's a lot more that could be done, but I need
to continue; I had wanted this done a week ago.
It is worth noting, though, that this finally achieves something I had been
wondering about since the inception of this project---how I'd represent
templates on the graph. I think this worked out rather nicely. It wasn't
even until a few months ago that I decided to use AIR instead of NIR for
that purpose (NIR wouldn't have worked).
And note how I didn't have to touch the program derivation at all---the
system test just works with the AIR change, because of the consistent
construction of the graph. Beautiful.
DEV-13708
This hoists the errors back into `AirAggregate`; I need dead states for the
`AirTplAggregate` parser so that it will know when to (and not to) interpret
tokens in the context of the template itself.
In a previous commit message, I had pondered whether it may be possible to
eliminate the dead state transition, and yet here I've used it with both of
the sub-parsers now. So it seems like the better option in the future may
be to narrow the type further---to say precisely _what_ types of tokens may
yield a dead state transition; otherwise you lose the match information from
the parser that yielded it.
A stubbornly persistent problem in Rust, this magical and hidden match
knowledge.
DEV-13708
This sets us up to be able to determine how `Dangling` expressions will be
rooted into templates.
This new strategy isn't yet handling `Dangling`; I wanted to get this
committed first so that the `Dangling` refactoring is more clear.
DEV-13708
Expressions were previously tied to packages. This prepares for using a
`Tpl` as a container for expressions.
This does not yet handle the situation of auto-rooting dangling expressions
within the container.
DEV-13708
This result in less useful debug output, but it'll be needed for using
a (possibly-anonymous) template as evidence.
This evidence is simply for debugging, and to require some sort of value
during development to help obviate when maybe something is being done
incorrectly (if no obvious value exists).
DEV-13708
This is more of the same refactoring that has been happening. This
extraction also helps emphasize the relationship between imported objects,
and isolates the growing number of test cases. This parser will only grow.
DEV-13708
Just as was done with the expression parser, which this will utilize. This
initializes it, but doesn't yet make use of it (`AirExprAggregate`).
Refactoring was definitely needed; decomposing this is quite a bit of work,
in no small part because of the complexity. This helps significantly.
DEV-13708
Mike Gerwitz