Things are starting to get interesting, and this shows how caching
information about template shape (rather than having to query the graph any
time we want to discover it) makes it easy to compose shapes.
This does not yet handle the unknown case. Before I do that, I'll want to
do some refactoring to address duplication in the `tpl` module.
DEV-13163
This enforces the new constraint that templates expanding into an `Expr`
context must only inline a single `Expr`.
Perhaps in the future we'll support explicit splicing, like `,@` in
Lisp. But this new restriction is intended for two purposes:
- To make templates more predictable (if you have a list of expressions
inlined then they will act differently depending on the type of
expression that they are inlined into, which means that more defensive
programming would otherwise be required); and
- To make expansion easier, since we're going to have to set aside an
expansion workspace ahead of time to ensure ordering (Petgraph can't
replace edges in-place). If we support multi-expansion, we'd have to
handle associativity in all expression contexts.
This'll become more clear in future commits.
It's nice to see all this hard work coming together now, though; it's easy
now to perform static analysis on the system, and any part of the graph
construction can throw errors with rich diagnostic information and still
recover properly. And, importantly, the system enforces its own state, and
the compiler helps us with that (the previous commits).
DEV-13163
This change is the first to utilize matching on edges to determine the state
of the template (to begin to derive its shape).
But this is notable for my finally caving on `min_specialization`.
The commit contains a bunch of rationale for why I introduced it. I've been
sitting on trying it for _years_. I had hoped for further progress in
determining a stabalization path, but that doesn't seem to be happening.
The reason I caved is because _not_ using it is a significant barrier to
utilizing robust types in various scenarios. I've been having to work
around that with significant efforts to write boilerplate code to match on
types and branch to various static paths accordingly. It makes it really
expensive to make certain types of changes, and it make the code really
difficult to understand once you start to peel back abstractions that try to
hide it.
I'll see how this goes and, if it goes well, begin to replace old methods
with specialization.
See the next commit for some cleanup. I purposefully left this a bit of a
mess (at the bottom of `asg::graph::object::tpl`) to emphasize what I'm
doing and why I introduced it.
DEV-13163
I'm not sure how I overlooked this previously, and I didn't notice until
trying to generate xmli output. I think I distracted myself with the
use of dangling status, which was not appropriate, and that has since
changed so that we have a dedicated concept.
This introduces the term "instantiation", or more specifically "lexical
instantiation". This is more specific and meaningful than simply
"expansion", which is what occurs during instantiation. I'll try to adjust
terminology and make things more consistent as I go.
DEV-13163
This makes `SPair` construction more concise, getting rid of the `into`
invocations. For now I have only made this change in AIR's tests, since
that's what I'm working on and I want to observe how this convention
evolves. This may also encourage other changes, e.g. placing spans within
the `toks` array, rather than having to jump around the test for them.
The comment for `spair` mentions why this is a test-only function. But it
also shows how dangerous `impl Into<SymbolId> for &str` can be, since it
seems so innocuous---it uses a global interner. I'll be interested to see a
year from now if I decided to forego that impl in favor of explicit
internment, since I'm not sure it's worth the convenience anymore.
DEV-13163
This has been bothering me for quite a long time, and is just more test
cleanup before I introduce more. I suspect this came from habit with the
previous Rust edition where `into_iter()` on arrays was a much more verbose
operation.
To be clear: this change isn't for performance. It's about not doing
something silly when it's unnecessary, which also sets a bad example for
others.
There are many other tests in other modules that will need updating at some
point.
DEV-13163
This produces a representation of abstract identifiers on the graph, for
`Expr`s at least. The next step will probably be to get this working
end-to-end in the xmli output before extending it to the other remaining
bindable contexts.
DEV-13163
This is intended to support NIR's lexical interpolation, which expands in
place into metavariables.
This commit does not yet contain the NIR portion (or xmli system test)
because Meta needs to be able to handle concatenation first; that's next.
DEV-13163
This introduces template/param and regenerates it in the xmli output. Note
that this does not check that applications reference known params; that's a
later phase.
DEV-13163
This change means that `asg::air` is now the only module that directly
invokes index-related methods on `Asg`. This clears the way, finally, to
removing the index from `Asg` entirely.
Not only does this result in a less awkward architecture, it also ensures
that lookups are forced to go through the system that understands and
controls lexical scoping, which will be able to give the correct answer.
Of course, the caveat is that the "correct" answer depends on what's
currently on the stack, depending on what type of lookup is being performed,
but those details are still encapsulated within the `asg::air` module and
its tests.
DEV-13162
The previous commit introduced a duplicate `asg_from_toks`; this just makes
it available publicly for any tests that might utilize AIR to lower the
barrier to writing such tests and provide some guidance in doing so.
DEV-13162
This requires the name as part of the package definition, which in turn
removes a state (and all the combinations resulting from it) from
AirAggregate, which results in significant complexity reduction for a very
complex part of the system.
Pushing this complexity outward results in a reduction of overall
complexity, and obviates the question of where NIR will receive a generated
name.
DEV-13162
This reverts commit da7fe96254e425bc7b75f8cf454465b71e27e372.
I'm a fool---this would be pursuant to a future plan that removes AirIdent
opaque tokens. But for now, I need it on IdentDecl and others, which
currently has a `Source` (that I want to go away, as just mentioned), which
contains the same information.
So maybe more to come on this...
DEV-13162
This allows for a canonical package name to be optionally provided to
explicitly resolve a reference against, avoiding a lexical lookup.
This change doesn't actually utilize this new value yet; it just
retains BC. The new argument will be used for the linker, since it already
knows the package that defined an identifier while reading the object file's
symbol table. It will also be used by tamec for the same purposes while
processing package imports.
DEV-13162
-- squashed with --
tamer: asg::air::ir::RefIdent: CanonicalName=SPair
The use of CanonicalName created an asymmetry between RefIdent and
BindIdent. The hope was to move CanonicalName instantiation outside of AIR
and into NIR, but doing so would be confusing and awkward without doing
something with BindIdent.
I don't have the time to deal with that for now, so let's observe how the
system continues to evolve and see whether hoisting it out makes sense in the
end. For now, this works just fine and I need to move on with the actual
goal of finishing package imports so that I can expand templates.
DEV-13162
I've been torturing myself trying to figure out how I want to generalize
indexing, lookups, and value numbering in a way that is appropriate for this
project (that is, not over-engineered relative to my needs).
Before I can do much of anything, though, I need to stop having indexing
only as a `Root` thing (previously it wasn't even tied to `Root`). This
makes that change for tamec, but temporarily removes scoping concerns until
I can add more specific types of indexing.
Not only does this allow cleaning up some `Ident`-specific stuff from `Asg`,
but the cleanup also helps to show that portions of the system aren't still
using Root-based globals.
The linker (`tameld`) still uses the old `global` methods for now; those
will eventually go away, but this needs to change to unify both tamec and
tameld once we get to imports as part of the compiler.
DEV-13162
This is needed to then support `@desc` for shorthand desugaring; it's
required by the XSLT-based compiler (and will eventually be required by
TAMER too).
DEV-13708
This recognizes template application within expressions. Since expressions
can occur within templates, this can occur arbitrarily deeply.
And with that, we have the core of the template system represented on the
graph. Of course, there are some glaring scoping issues to be resolved, but
those aren't unique to template application.
DEV-13708
I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
`AirAggregate` now handles all delegation to `AirExprAggregate`. This is
possible because `AirAggregate` is now the superstate for each of these
parsers, so `AirTplAggregate` is able to transition to a state that is not
its own.
This does not go so far as reaching the ultimate objective---having nested
template support---even though it'd be fairly simple to do now; there's
going to be a number of interesting consequences to these changes, and a bit
of cleanup is still needed, and I want tests observing this functionality to
accompany those changes. That is: let's keep this a refactoring, to the
extent that it's possible.
Things are getting much easier to understand now, and much cleaner.
DEV-13708
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least). And, as you can see, there's still a lot of TODO items here.
I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.
But, it works.
DEV-13708
I'm not happy with this implementation. The linear search is undesirable,
but not too bad (and maybe wouldn't even be worth caching, if this were the
whole story), but we _also_ need to prevent duplicate identifiers. We are
not going to want to perform a linear search of a linked list (effectively)
every time we add an identifier to check for uniqueness, so I think the
caching is going to have to be generalized very shortly anyway.
As it stands now, a duplicate identifier would cause an error at expansion
time. That's not what we want, but it's not terrible, because you can have
that same problem in normal circumstances without local conflicts.
But this'll be used for metavariables as well, where we absolutely _do_ want
to fail at template definition time.
DEV-13708
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.
Templates must not contribute to this environment until expansion. Further,
metavariables will not be present in this environment. To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed. This will also allow for the creation of more general
methods down the line.
DEV-13708
This makes the tests quite a bit easier to understand visually. I've been
doing this with all new tests but had to go back to some old ones, and still
have more to go back to. Baby steps.
DEV-13708
This this a big change that's difficult to break up, and I don't have the
energy after it.
This introduces nullary template application, short- and long-form. Note
that a body of the short form is a `@values@` argument, so that's not
supported yet.
This continues to formalize the idea of what "template application" and
"template expansion" mean in TAMER. It makes a separate `TplApply`
unnecessary, because now application is simply a reference to a
template. Expansion and application are one and the same: when a template
expands, it'll re-bind metavariables to the parent context. So in a
template context, this amounts to application.
But applying a closed template will have nothing to bind, and so is
equivalent to expansion. And since `Meta` objects are not valid outside of
a `Tpl` context, applying a non-closed template outside of another template
will be invalid.
So we get all of this with a single primitive (getting the "value" of a
template).
The expansion is conceptually like `,@` in Lisp, where we're splicing trees.
It's a mess in some spots, but I want to get this committed before I do a
little bit of cleanup.
See the Air docblock for more information. I'm introducing new tokens for
the template system, which uses the terms "free" and "closed". I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.
DEV-13708
The intent was to have a very simple implementation of `hold_dangling` and
have everything work. But, I had a nasty surprise when the system tests
caught bug caused by some interesting depth interactions as it relates to
`xmli` and auto-closing.
I added an extra test/example in `asg::graph::visit::test` to illustrate the
situation; it was difficult to derive from the traces, but trivially obvious
once I wrote it out as an example.
With that, templates can now aggregate tokens for dangling expressions.
DEV-13708
And finally we have tokens aggregated onto the ASG in the context of a
template. I expected to arrive here much more quickly, but there was a lot
of necessary refactoring. There's a lot more that could be done, but I need
to continue; I had wanted this done a week ago.
It is worth noting, though, that this finally achieves something I had been
wondering about since the inception of this project---how I'd represent
templates on the graph. I think this worked out rather nicely. It wasn't
even until a few months ago that I decided to use AIR instead of NIR for
that purpose (NIR wouldn't have worked).
And note how I didn't have to touch the program derivation at all---the
system test just works with the AIR change, because of the consistent
construction of the graph. Beautiful.
DEV-13708
Mike Gerwitz