Also known as metavariables or template parameters.
This is a bit of a tortured excursion, trying to figure out how I want to
best represent this. I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.
`AirTpl::TplApply` highlights some remaining questions. What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such. But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue. But I'd rather address that head-on in
the future.
So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time. This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.
My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them. That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.
But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems
viable.
DEV-13708
See the Air docblock for more information. I'm introducing new tokens for
the template system, which uses the terms "free" and "closed". I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.
DEV-13708
This removes special cases, but it does complicate the parent `AirAggregate`
parser. A pattern of delegation is forming, though abstracting it may be an
interesting challenge, given Rust's limitation on macro invocations as match
arms. But, I think I can manage by generating the entire match using a
macro with a match-compatible syntax, augmenting where
needed...maybe. This'll be messy.
...but if I can write the nightmare that is `ele_parse!`, I'm sure I can
manage this. I just prefer to avoid complex macros unless I really need
them.
DEV-13708
Now that these are actually intended to be used as part of the build, this
is a more appropriate location. I originally wrote it as a manual tool.
DEV-13708
This parses the declarative `object_rel!` definitions from the Rust sources
and produces a DOT representation of the ontology of the graph, which can
then be rendered using Graphviz.
This does not yet introduce it into the build; it ought to be run as part of
`make check` (without rendering with Graphviz) to ensure that we catch
breaking changes, and `make html` ought to integrate it into the
documentation, perhaps as part of `asg::graph` or `asg::graph::object`.
DEV-13708
Small break from templates for something easier. I have COVID-19, so I'll
use that as my excuse for wanting to be more lazy.
The real reason is to see some more concrete progress and ensure that
patterns hold for simple expressions before further refactoring.
But, before I proceed with such refactoring, I really ought to approach
something that requires a NIR desugaring step, like case statements.
DEV-13708
Going higher than that doesn't make sense because we're in shell and
invoking commands all around this, so even milliseconds isn't going to be
entirely accurate here. However, what I am more interested in is observing
time relative to other runs; this isn't intended for profiling, but for
eyeballing unexpected behavior.
DEV-13708
There's a lot to look at, especially in the event of failure. Further, I
wanted to add additional statistics that could be eyeballed.
Right now, tamec is too fast (at least on my machine) for the precision of
/usr/bin/time: we need milliseconds, but we only get hundredths of a
second. So it'll all show as 0:00.00s. Which is okay, for now; it just
shouldn't exceed that. ;)
DEV-13708
The intent was to have a very simple implementation of `hold_dangling` and
have everything work. But, I had a nasty surprise when the system tests
caught bug caused by some interesting depth interactions as it relates to
`xmli` and auto-closing.
I added an extra test/example in `asg::graph::visit::test` to illustrate the
situation; it was difficult to derive from the traces, but trivially obvious
once I wrote it out as an example.
With that, templates can now aggregate tokens for dangling expressions.
DEV-13708
This won't try the fixpoint test if the prior one fails, which will always
cause that one to fail. And it further won't attempt the diff on
compilation failure.
DEV-13708
And finally we have tokens aggregated onto the ASG in the context of a
template. I expected to arrive here much more quickly, but there was a lot
of necessary refactoring. There's a lot more that could be done, but I need
to continue; I had wanted this done a week ago.
It is worth noting, though, that this finally achieves something I had been
wondering about since the inception of this project---how I'd represent
templates on the graph. I think this worked out rather nicely. It wasn't
even until a few months ago that I decided to use AIR instead of NIR for
that purpose (NIR wouldn't have worked).
And note how I didn't have to touch the program derivation at all---the
system test just works with the AIR change, because of the consistent
construction of the graph. Beautiful.
DEV-13708
This hoists the errors back into `AirAggregate`; I need dead states for the
`AirTplAggregate` parser so that it will know when to (and not to) interpret
tokens in the context of the template itself.
In a previous commit message, I had pondered whether it may be possible to
eliminate the dead state transition, and yet here I've used it with both of
the sub-parsers now. So it seems like the better option in the future may
be to narrow the type further---to say precisely _what_ types of tokens may
yield a dead state transition; otherwise you lose the match information from
the parser that yielded it.
A stubbornly persistent problem in Rust, this magical and hidden match
knowledge.
DEV-13708
This sets us up to be able to determine how `Dangling` expressions will be
rooted into templates.
This new strategy isn't yet handling `Dangling`; I wanted to get this
committed first so that the `Dangling` refactoring is more clear.
DEV-13708
Expressions were previously tied to packages. This prepares for using a
`Tpl` as a container for expressions.
This does not yet handle the situation of auto-rooting dangling expressions
within the container.
DEV-13708
This result in less useful debug output, but it'll be needed for using
a (possibly-anonymous) template as evidence.
This evidence is simply for debugging, and to require some sort of value
during development to help obviate when maybe something is being done
incorrectly (if no obvious value exists).
DEV-13708
This is more of the same refactoring that has been happening. This
extraction also helps emphasize the relationship between imported objects,
and isolates the growing number of test cases. This parser will only grow.
DEV-13708
Just as was done with the expression parser, which this will utilize. This
initializes it, but doesn't yet make use of it (`AirExprAggregate`).
Refactoring was definitely needed; decomposing this is quite a bit of work,
in no small part because of the complexity. This helps significantly.
DEV-13708
This works around limitations of Rust's borrow checker as of the time of
writing. See the provided documentation for more information.
The branch context is not yet exposed to the `delegate` family of methods;
it will be added only as needed in the future.
DEV-13708
This delegates expression parsing to `AirExprAggregate`, in an effort to
both begin to simplify the understanding and maintenance of `AirAggregate`;
and allow for parser composition for template parsing.
This utilizes the prior changes for token sum types to precisely define the
subset of AIR tokens supported by the expression parser. This differs from
prior approaches which delegated until a dead state, relying on runtime
information to determine if a parser has finished. This allows us to
determine that statically.
I do want to be able to eliminate the dead state from the parser so we can
get rid of the `unreachable!`, but I need to move on; that's something I had
tried to do in the past too, which ended up adding a bit of complexity, and
I'll have to consider my options in the future, including whether the dead
state transition can be entirely eliminated in favor of the combination of
these sum types and recovery; the parsing framework decisions were made
while recovery was still an open question, at least in practice.
DEV-13708
This was a rather frustrating thing to encounter. I was working on
refactoring `AirAggregate`, and found that my tests were hanging despite no
apparent cause in the parser itself.
As it turns out, rather than failing with a `FinalizeError` as I
expected (since I was mid-refactor), `collect()` was allocating space for an
endless stream of errors. This was easily verified by adding a `take(x)`
and observing the assertion failure (in this case, in `close_pkg_mid_expr`).
This happens to be the first time in a long time that I actually had to
debug---the combination of robust types as proofs and tests to fill in the
gaps means that runtime issues are caught at build time in all but
exceptional cases (like this one).
It's also worth noting that, because of my policy of iterating only at the
higher levels of the program, it was clear that this must somehow be
Parser-related, since that's the only part of the system that has the
potential for unbounded recursion due to its cyclic state machines.
DEV-13708
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
This sets the stage for template parsing, and finally decides how we're
going to represent templates on the ASG. This is going to start simple,
since my original plans for improving how templates are
handled (conceptually) is going to have to wait.
This is the last difficult object type to figure out, with respect to graph
representation and derivation, so I wanted to get it out of the way.
DEV-13708
I wasn't initially sure whether I'd want separate tokens for different types
of identifying operations, but now that I see that it is clear from the
current state of the parser, there's no need.
This matches the name of the token in NIR.
DEV-13708
The previous commit demonstrated the amount of boilerplate necessary for
introducing new `ObjectKind`s; this abstracts away a lot of that
boilerplate, and allows for declarative relationship definition for the
ASG's ontology.
DEV-13708
There's quite a bit of boilerplate here that'll eventually need factoring
out. But it's also clear that it is somewhat onerous to add new object
types.
Note that a good chunk of this burden is _intentional_, via exhaustiveness
checks---adding a new type of object is an exceptional occurrence (well, in
principle, but we haven't added them all yet, so it'll be more common
initially), and we'd rather be safe to ensure that everything is properly
considering how that new type of object interacts with it.
Let's not confuse coupling with safety---the latter causes a burden because
of the former, not because of itself; it provides a service to us.
But, nonetheless, we'll want to reduce this burden somewhat since there are
a number more to add.
DEV-13708
Just as `rate` is a `sum`, `classify` is an `all` by default. The `@any`
attribute will change that interpretation, though I only intend to recognize
that in parsing later on, not emit that in XMLI.
DEV-13708
Let's start to be explicit about what's missing as we continue to add new
tokens; the exhaustiveness checks throughout the system will guide the
changes that need to be made.
DEV-13708
The element only, no attributes yet.
I'll keep forming boilerplate until abstraction points become obvious with
more variety; this is still pretty close to what was already supported.
DEV-13708
We already had `TreeContext`, and I'm passing the same arguments around, so
this uses it to lift arguments out of these functions, like partial
application.
DEV-13708
This tidies this method up into a decent state that I'm fairly content
with. This goes to emphasize my dislike of returns, which muddies control
flow and makes the code more difficult to read at a glance, which increase
the likelihood of logic bugs.
`match` statements in tail position, on the other hand, are very clear, and
less cognitively burdensome since you can see each individual code path at a
glance.
DEV-13708
This begins to develop a pattern for doing these transformations. I had
tried a number of things using iterators, but I wasn't satisfied with either
how they were turning out; had to fight too much with the type system; or
had to resort to heap allocations. Sticking with an explicit
`push`/`push_all` for now works just fine.
Almost done cleaning up `AsgTreeToXirf::parse_token`, and then I can move on
to introducing more objects.
DEV-13708
This is generic over the source, just as the target, defaulting just the
same to `ObjectIndex`.
This allows us to use only the edge information provided rather than having
to perform another lookup on the graph and then assert that we found the
correct edge. In this case, we're dealing with an `Ident->Expr` edge, of
which there is only one, but in other cases, there may be many such edges,
and it wouldn't be possible to know _which_ was referred to without also
keeping context of the previous edge in the walk.
So, in addition to avoiding more indirection and being more immune to logic
bugs, this also allows us to avoid states in `AsgTreeToXirf` for the purpose
of tracking previous edges in the current path. And it means that the tree
walk can seed further traversals in conjunction with it, if that is so
needed for deriving sources.
More cleanup will be needed, but this does well to set us up for moving
forward; I was too uncomfortable with having to do the separate
lookup. This is also a more intuitive API.
But it does have the awkward effect that now I don't need the pair---I just
need the `Object`---but I'm not going to remove it because I suspect I may
need it in the future. We'll see.
The TODO references the fact that I'm using a convenient `resolve_oi_pairs`
instead of resolving only the target first and then the source only in the
code path that needs it. I'll want to verify that Rust will properly
optimize to avoid the source resolution in branches that do not need it.
DEV-13708
This makes the inner `Object` type generic (but defaulting to the same inner
types as before) so that it can be used as a sum type for various types
where `ObjectKind`-based narrowing is required.
In this case, it's used to narrow `ObjectIndex` alongside the inner
`ObjectKind` so that the two are definitely in sync. This not only results
in cleaner code and a more intuitive API that's approachable to people
less familiar with the system, but it also helps to eliminate logic bugs
that might result form manually narrowing (as was done before this change).
DEV-13708
This was a fairly simple addition, since rate blocks already lower into sum
expressions; these are just non-identified.
This does emphasize that the nir::parse `ele_parse!` abstraction I spent so
much time on ended up not being a perfect fit, as it now has some
boilerplate after it was stripped of much of its capabilities some time ago.
Don't worry, `nir::air` and `asg::graph::xmli` will get cleaned up.
DEV-13708
This extends the POC a bit by beginning to reconstruct rate blocks (note
that NIR isn't producing sub-expressions yet).
Importantly, this also adds the first system tests, now that we have an
end-to-end system. This not only gives me confidence that the system is
producing the expected output, but serves as a compromise: writing unit or
integration tests for this program derivation would be a great deal of work,
and wouldn't even catch the bugs I'm worried most about; the lowering
operation can be written in such a way as to give me high confidence in its
correctness without those more granular tests, or in conjunction with unit
or integration tests for a smaller portion.
DEV-13708
Mike Gerwitz