Historically, the ASG was better described as a "dependency graph",
containing only identifiers (which are simply called "symbols" in the
XSLT-based compiler). Consequently, it was appropriate for the graph to
have operations specific to identifiers. (Indeed, that's the only type of
object the graph supported.)
Much has changed since then. This cleans things up, and makes parenting
identifiers to root an _explicit_ operation. This will make it easier to
move forward with handling of scope, and importing identifiers into
packages, and removing `Source`, and so on.
DEV-13162
I've been torturing myself trying to figure out how I want to generalize
indexing, lookups, and value numbering in a way that is appropriate for this
project (that is, not over-engineered relative to my needs).
Before I can do much of anything, though, I need to stop having indexing
only as a `Root` thing (previously it wasn't even tied to `Root`). This
makes that change for tamec, but temporarily removes scoping concerns until
I can add more specific types of indexing.
Not only does this allow cleaning up some `Ident`-specific stuff from `Asg`,
but the cleanup also helps to show that portions of the system aren't still
using Root-based globals.
The linker (`tameld`) still uses the old `global` methods for now; those
will eventually go away, but this needs to change to unify both tamec and
tameld once we get to imports as part of the compiler.
DEV-13162
This supports arbitrary documentation as sibling text (mixed content, in XML
terms). The motivation behind this change is to permit existing system
tests to succeed when `Todo | TodoAttr` are both rejected, rather than
having to ignore this.
TAME has always had a philosophy of literate documentation, however it was
never fully realized. This just maintains the status quo; the text is
unstructured, and maybe will be parsed in the future.
Unfortunately, this does _not_ include the output in the `xmli` file or the
system tests. The reason has nothing to do with TAMER---`xmllint` does not
format the output when there is mixed content, it seems, and I need to move
on for now; I'll consider my options in the future. But, it's available on
the graph and ready to go.
DEV-13708
This introduces a new `Doc` object that can be owned by `Expr` (only atm)
and contain what it describes as a concise independent clause. This
construction is not enforced, and is only really obvious today via the
Summary Pages.
There's a lot of latent and unrealized potential in TAME's documentation
philosophy that was never realized, so this will certainly evolve over
time. But for now, the primary purpose was to get `@desc` working on things
like classifications so that `xmli` output can compile for certain
packages.
DEV-13708
This doesn't do the actual hard work yet of resolving and loading a package,
but it does place it on the graph and re-derive it into the xmli output.
DEV-13708
This recognizes template application within expressions. Since expressions
can occur within templates, this can occur arbitrarily deeply.
And with that, we have the core of the template system represented on the
graph. Of course, there are some glaring scoping issues to be resolved, but
those aren't unique to template application.
DEV-13708
I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
This eliminates the TODOs that existed when looking for an OI for rooting an
identifier.
The change to `rooting_ci` is ridiculous, but I want to get other things
done before I jump down the rabbit hole of generalizing that (indexing local
identifiers). Though I have an approach in mind.
DEV-13708
Just some continued cleanup.
Unfortunately, we have sacrificed knowing a package OI must exist
statically, even though one will always be available.
DEV-13708
This has AirAggregate preempt Expr parsing in the same way as templates,
rather than having `AirTplAggregate` concern itself with expression
tokens. This continues to simplify `AirTplAggregate`, which was getting
quite complex not too long ago.
A pattern is now emerging for the call/ret convention for preemption. That
was intentional, but it's nice to see it manifest so obviously before I
abstract it away.
DEV-13708
This was extracted from xir::parse::ele in previous commits. The
conventions help to ensure that pushing and returns are being performed
correctly. The abstraction will continue to evolve.
This ends up using `Ready` as the dead state. I need to determine if this
is ideal, and if so, maybe just use `Default`, otherwise yield an error.
DEV-13708
`AirAggregate` now handles all delegation to `AirExprAggregate`. This is
possible because `AirAggregate` is now the superstate for each of these
parsers, so `AirTplAggregate` is able to transition to a state that is not
its own.
This does not go so far as reaching the ultimate objective---having nested
template support---even though it'd be fairly simple to do now; there's
going to be a number of interesting consequences to these changes, and a bit
of cleanup is still needed, and I want tests observing this functionality to
accompany those changes. That is: let's keep this a refactoring, to the
extent that it's possible.
Things are getting much easier to understand now, and much cleaner.
DEV-13708
I love deleting code I just wrote...
This doesn't solve the underlying problems with identifiers, but it does at
least lift it into the `AirAggregateCtx`, allowing `AirExprAggregate` to be
even further simplified. Now the `From` implementation is not specialized
and we can readily convert to a SuperState.
There's still a lot of TODOs here, though. And some of them will
unfortunately require runtime checks where there was previously a
compile-time check. But that's okay in a lot of the cases, because the
empty behavior will replace existing error checks.
DEV-13708
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
This is more of the same of the previous commit, but in a more digestable
chunk. We now have child states that are able to be constructed using a
simple `From`, which is important to making `AirAggregate` a `SuperState`.
This also makes `AirStack` act like a prototype chain for `ObjectIndex`es,
creating environments where context shadows. The linear search should only
have to check the last two frames (e.g. an Expr has a parent Pkg or Tpl
context which will have a `rooting_oi` value), and this is only done during
a rooting operation.
DEV-13708
This begins to introduce `AirStack` and starts to migrate context away from
the individual `ParseState`s onto the stack.
I should have started to commit earlier; this is getting a bit large and
makes it hard to follow what I'm doing so, hopefully stopping a little bit
short will allow the following commit to show that.
This is a work-in-progress change. All tests pass, but the refactoring is
incomplete. The `AirStack` abstraction is _also_ incomplete and will have
better, more domain-specific operations that make it harder to mess up
pairing pushes with pops.
The purpose of doing this is to allow `AirAggregate` to serve exclusively as
a sum state, which can then become a SuperState, much like `ele_parse!`'s
approach.
The _end_ goal of all of this is arbitrary template nesting.
DEV-13708
To simplify things in support of upcoming changes, we'll just instantiate a
new one as needed. This doesn't have an appreciable performance impact, so
the optimization is premature. It was done just because it was more of the
same that TAMER was already doing, but now it's making things more
difficult.
DEV-13708
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
Previously, `AirTplAggregate` worked only in a `Pkg` context, being able to
root `Tpl` `Ident`s in `Pkg` and expand only into `Pkg`. This still does
the same, but generalizes to allow for different roots and expansion
targets.
This will be utilized to parse nested templates.
DEV-13708
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.
Templates must not contribute to this environment until expansion. Further,
metavariables will not be present in this environment. To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed. This will also allow for the creation of more general
methods down the line.
DEV-13708
Also known as metavariables or template parameters.
This is a bit of a tortured excursion, trying to figure out how I want to
best represent this. I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.
`AirTpl::TplApply` highlights some remaining questions. What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such. But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue. But I'd rather address that head-on in
the future.
So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time. This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.
My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them. That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.
But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems
viable.
DEV-13708
See the Air docblock for more information. I'm introducing new tokens for
the template system, which uses the terms "free" and "closed". I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.
DEV-13708
This removes special cases, but it does complicate the parent `AirAggregate`
parser. A pattern of delegation is forming, though abstracting it may be an
interesting challenge, given Rust's limitation on macro invocations as match
arms. But, I think I can manage by generating the entire match using a
macro with a match-compatible syntax, augmenting where
needed...maybe. This'll be messy.
...but if I can write the nightmare that is `ele_parse!`, I'm sure I can
manage this. I just prefer to avoid complex macros unless I really need
them.
DEV-13708
And finally we have tokens aggregated onto the ASG in the context of a
template. I expected to arrive here much more quickly, but there was a lot
of necessary refactoring. There's a lot more that could be done, but I need
to continue; I had wanted this done a week ago.
It is worth noting, though, that this finally achieves something I had been
wondering about since the inception of this project---how I'd represent
templates on the graph. I think this worked out rather nicely. It wasn't
even until a few months ago that I decided to use AIR instead of NIR for
that purpose (NIR wouldn't have worked).
And note how I didn't have to touch the program derivation at all---the
system test just works with the AIR change, because of the consistent
construction of the graph. Beautiful.
DEV-13708
This hoists the errors back into `AirAggregate`; I need dead states for the
`AirTplAggregate` parser so that it will know when to (and not to) interpret
tokens in the context of the template itself.
In a previous commit message, I had pondered whether it may be possible to
eliminate the dead state transition, and yet here I've used it with both of
the sub-parsers now. So it seems like the better option in the future may
be to narrow the type further---to say precisely _what_ types of tokens may
yield a dead state transition; otherwise you lose the match information from
the parser that yielded it.
A stubbornly persistent problem in Rust, this magical and hidden match
knowledge.
DEV-13708
This sets us up to be able to determine how `Dangling` expressions will be
rooted into templates.
This new strategy isn't yet handling `Dangling`; I wanted to get this
committed first so that the `Dangling` refactoring is more clear.
DEV-13708
Expressions were previously tied to packages. This prepares for using a
`Tpl` as a container for expressions.
This does not yet handle the situation of auto-rooting dangling expressions
within the container.
DEV-13708
This is more of the same refactoring that has been happening. This
extraction also helps emphasize the relationship between imported objects,
and isolates the growing number of test cases. This parser will only grow.
DEV-13708
Just as was done with the expression parser, which this will utilize. This
initializes it, but doesn't yet make use of it (`AirExprAggregate`).
Refactoring was definitely needed; decomposing this is quite a bit of work,
in no small part because of the complexity. This helps significantly.
DEV-13708
This works around limitations of Rust's borrow checker as of the time of
writing. See the provided documentation for more information.
The branch context is not yet exposed to the `delegate` family of methods;
it will be added only as needed in the future.
DEV-13708
This delegates expression parsing to `AirExprAggregate`, in an effort to
both begin to simplify the understanding and maintenance of `AirAggregate`;
and allow for parser composition for template parsing.
This utilizes the prior changes for token sum types to precisely define the
subset of AIR tokens supported by the expression parser. This differs from
prior approaches which delegated until a dead state, relying on runtime
information to determine if a parser has finished. This allows us to
determine that statically.
I do want to be able to eliminate the dead state from the parser so we can
get rid of the `unreachable!`, but I need to move on; that's something I had
tried to do in the past too, which ended up adding a bit of complexity, and
I'll have to consider my options in the future, including whether the dead
state transition can be entirely eliminated in favor of the combination of
these sum types and recovery; the parsing framework decisions were made
while recovery was still an open question, at least in practice.
DEV-13708
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
This sets the stage for template parsing, and finally decides how we're
going to represent templates on the ASG. This is going to start simple,
since my original plans for improving how templates are
handled (conceptually) is going to have to wait.
This is the last difficult object type to figure out, with respect to graph
representation and derivation, so I wanted to get it out of the way.
DEV-13708
I wasn't initially sure whether I'd want separate tokens for different types
of identifying operations, but now that I see that it is clear from the
current state of the parser, there's no need.
This matches the name of the token in NIR.
DEV-13708
The `Pkg` span will now properly reflect the entire definition of the
package including the opening and closing tags.
This was found while I was working on a graph traversal.
DEV-13597
I noticed this while working on a graph traversal. The unit test used the
same span for both the reference _and_ the binding, so I didn't notice. -_-
The problem with this, though, is that we do not have a separate span
representing the source location of the identifier reference. The reason is
that we decided to re-use an existing node rather than creating another one,
which would add another inconvenient layer of indirection (and complexity).
So, I may have to add (optional?) spans to edges.
DEV-13708
This causes a package definition to be rooted (so that it can be easily
accessed for a graph walk). This keeps consistent with the new
`ObjectIndex`-based API by introducing a unit `Root` `ObjectKind` and the
boilerplate that goes with it.
This boilerplate, now glaringly obvious, will be refactored at some point,
since its repetition is onerous and distracting.
DEV-13159
Included in this diff are the corresponding changes to the graph to support
the change. Adding the edge was easy, but we also need a way to get the
package for an identifier. The easiest way to do that is to modify the edge
weight to include not just the target node type, but also the source.
DEV-13159
This does not yet create edges from identifiers to the package; just getting
this introduced was quite a bit of work, so I want to get this committed.
Note that this also includes a change to NIR so that `Close` contains the
entity so that we can pattern-match for AIR transformations rather than
retaining yet another stack with checks that are already going to be done by
AIR. This makes NIR stand less on its own from a self-validation point, but
that's okay, given that it's the language that the user entered and,
conceptually, they could enter invalid NIR the same as they enter invalid
XML (e.g. from a REPL).
In _practice_, of course, NIR is lowered from XML and the schema is enforced
during that lowering and so the validation does exist as part of that
parsing.
These concessions speak more to the verbosity of the language (Rust) than
anything.
DEV-13159
This adds support for identifier references, adding `Ident` as a valid edge
type for `Expr`.
There is nothing in the system yet to enforce ontology through levels of
indirection; that will come later on.
I'm testing these changes with a very minimal NIR parse, which I'll commit
shortly.
DEV-13597
This provides the initial implementation allowing an identifier to be
defined (bound to an object and made transparent).
I'm not yet entirely sure whether I'll stick with the "transparent" and
"opaque" terminology when there's also "declare" and "define", but a
`Missing` state is a type of declaration and so the distinction does still
seem to be important.
There is still work to be done on `ObjectIndex::<Ident>::bind_definition`,
which will follow. I'm going to be balancing work to provide type-level
guarantees, since I don't have the time to go as far as I'd like.
DEV-13597
This introduces a number of abstractions, whose concepts are not fully
documented yet since I want to see how it evolves in practice first.
This introduces the concept of edge ontology (similar to a schema) using the
type system. Even though we are not able to determine what the graph will
look like statically---since that's determined by data fed to us at
runtime---we _can_ ensure that the code _producing_ the graph from those
data will produce a graph that adheres to its ontology.
Because of the typed `ObjectIndex`, we're also able to implement operations
that are specific to the type of object that we're operating on. Though,
since the type is not (yet?) stored on the edge itself, it is possible to
walk the graph without looking at node weights (the `ObjectContainer`) and
therefore avoid panics for invalid type assumptions, which is bad, but I
don't think that'll happen in practice, since we'll want to be resolving
nodes at some point. But I'll addres that more in the future.
Another thing to note is that walking edges is only done in tests right now,
and so there's no filtering or anything; once there are nodes (if there are
nodes) that allow for different outgoing edge types, we'll almost certainly
want filtering as well, rather than panicing. We'll also want to be able to
query for any object type, but filter only to what's permitted by the
ontology.
DEV-13160
This addresses the two outstanding `todo!` match arms representing errors in
lowering expressions into the graph. As noted in the comments, these errors
are unlikely to be hit when using TAME in the traditional way, since
e.g. XIR and NIR are going to catch the equivalent problems within their own
contexts (unbalanced tags and a valid expression grammar respectively).
_But_, the IR does need to stand on its own, and I further hope that some
tooling maybe can interact more directly with AIR in the future.
DEV-13160
Mike Gerwitz