The number of files that had to be changed here emphasizes the need for a
better abstraction for indirection in the future. I just don't have the
time to do so yet.
This introduces the same type of indirection that was needed for abstract
bindings: we need to be able to distinguish between a metavariable
identifier's lexical value as a _string_ vs. a _reference_; the former is
how it was interpreted previously, which meant that interpolation would not
cause an edge to be created to the metavariable, which in turn means that
the metavariable was not referenced. This in turn blocked a change to
require that all metavariables be referenced.
I'm climbing back up my stash stack.
DEV-13163
This creates a new `AirDocAggregate` child parser and handled documentation
in a consistent way for all object types. The logic that was previously
object-specific now lives on the edge hooks that did not exist when doc
parsing was originally introduced.
This prepares for doc refs so that metavariables can be used for doc
generation (most notably for interpolated documentation strings in
templates).
This also introduces a new error type specifically for meta, giving guidance
to use `<text>`. I don't like that it builds in source language assumptions
in the help message provided, however I'd rather it be able to provide some
guidance to people, especially given that the XML source language is the
only one recognized by TAMER at the moment.
DEV-13163
Only internal modules ought to be able to mutate objects. This is important
now with the use of the `f::Map` trait, since traits don't allow for private
methods, which were previously the means of maintaining encapsulation.
DEV-13163
See included documentation for more information. This completes `MetaState`
inference, for now, until we are able to be notified when missing
identifiers acquire bindings.
DEV-13163
This moves us closer toward template expansion by allowing us to look at an
`Expr` and determine whether it requires any expansion at all. Naturally,
if something doesn't require expansion, we're able to duplicate it in the
tree without duplicating any data on the graph by simply adding a tree edge
to it. But, that's for a future commit; this just tracks the status.
Just as with some recent previous changes, we're not yet notifying `Expr`s
when an `Ident` is no longer `Missing`, and so order matters for now. This
will be rectified in the future.
DEV-13163
This prepares for the addition of more fields.
I don't want to sacrifice exhaustiveness checking, which is unfortunately
more verbose for structs when ignoring fields. This verbosity will be
adopted in certain situations, like shown in this patch, to ensure that
we're made aware of areas of the program that require our attention when
fields are added.
This also shows that I'll need a struct arm for `impl_mono_map!`.
DEV-13163
This removes the token of lookahead and just does what needs to be done in a
more clear manner. There is no room for interpretation in what this is
doing now, and `TplEnd` delegates to `close` just as this does.
DEV-13163
We are now able to have confidence that the graph is properly constructed
with or without a reference span depending on whether the edge is a tree or
cross edge. Consequently, the span is now a reliable indicator of whether
an edge is tree or cross in _all_ cases, not just in dynamic ones.
In fact, this did catch a couple cases where the spans were _not_ properly
applied.
This will in turn give me confidence moving forward with static analysis
based on the graph (and edge hooks).
I could go further than this by introducing a new span type in place of
`Option<Span>`, which would also allow me to get rid of having two methods
on `Asg`, but I want to move on for now; this can be cleaned up more later
on.
It's also worth noting that this explicit method-based distinction between
edge types also means that each caller will carefully consider how the
operation affects the graph. Previously, that consideration was framed very
differently: "do I need a contextual span or not?". That's not the right
question to ask.
DEV-13163
When this span was originally introduced for an edge, I wasn't sure whether
it may have use cases beyond just references. At this point, I'd rather be
explicit; it can evolve over time if need be.
There is nothing yet that prevents a span from being added to a tree edge,
though.
DEV-13163
This also hoists `?` out of the inner match arms to make control flow easier
to follow. I try to balance convenience with brutal explicitness, to make
control flow more obvious and make it easier to reason about what the system
is doing at every point in the program.
DEV-13163
This allows removing the `diagnostic_panic!` from the `Ident` edge handling
for `Tpl`.
I do need to create a generic sum type macro at some point, considering how
extensively these are used.
DEV-13163
Oh, identifiers. What a mess you make of the graph. As indirection tends
to do, I suppose.
This still needs some cleanup; I wanted to get something working first. In
particular:
- Common operations need refactoring;
- Having all this code in the edge definition makes for a mess;
- `Ident` references really ought not be returned by `Ident::definition`
since an identifier will never be a definition.
...I'm sure there's more.
DEV-13163
This renames the previous operation to `definition_narrow` and creates a new
`definition` that does not attempt to narrow. This is important for
distinguishing between a missing defintion and a definition of a given type,
and will be important now that we're beginning to reason about identifiers
that were resolved during parsing.
DEV-13163
I changed my mind from the previous commit---I do in fact want to inhibit
edge additions in some cases, to override what is added (e.g. to wrap a
template application in an Expr workspace).
The approach that I was trying to take was to override `from_oi` and return
it to the caller, but with all the dynamic edges, that ended up being a lot
more code than I feel was worth it; it was too complex. This is a much
simpler (and more flexible) approach---we simply carry out whatever graph
operations we want in `pre_add_edge` using the provided `asg` reference, and
then after we're done, simply omit `commit` on the original.
DEV-13163
This does two things:
1. Removes callback; it didn't add anything of practical value.
The operation will simply be performed as long as no error is provided
by the callee.
2. Consolodates three arguments into `ProposedRel`. This makes blocks in
`object_rel!` less verbose and boilerplate-y.
I'll probably implement `TplShape::Unknown` via the dynamic `Ident` `Tpl`
edge before continuing with any cleanup. This is getting pretty close to
reasonable for future implementations.
DEV-13163
This helps to remove some boilerplate. Testing this out in
`asg::graph::object::tpl` before applying it to other things; really `Map`
can just go away entirely then since it can be implemented in terms of
`TryMap`, but maybe it should stick around for manual impls (implementing
`TryMap` manually is more work).
DEV-13163
At least not how most people expect functors to be. I'm really just using
this as a map with powerful inference properties that make writing code more
pleasent.
And I need fallible methods now too.
DEV-13163
Things are starting to get interesting, and this shows how caching
information about template shape (rather than having to query the graph any
time we want to discover it) makes it easy to compose shapes.
This does not yet handle the unknown case. Before I do that, I'll want to
do some refactoring to address duplication in the `tpl` module.
DEV-13163
This enforces the new constraint that templates expanding into an `Expr`
context must only inline a single `Expr`.
Perhaps in the future we'll support explicit splicing, like `,@` in
Lisp. But this new restriction is intended for two purposes:
- To make templates more predictable (if you have a list of expressions
inlined then they will act differently depending on the type of
expression that they are inlined into, which means that more defensive
programming would otherwise be required); and
- To make expansion easier, since we're going to have to set aside an
expansion workspace ahead of time to ensure ordering (Petgraph can't
replace edges in-place). If we support multi-expansion, we'd have to
handle associativity in all expression contexts.
This'll become more clear in future commits.
It's nice to see all this hard work coming together now, though; it's easy
now to perform static analysis on the system, and any part of the graph
construction can throw errors with rich diagnostic information and still
recover properly. And, importantly, the system enforces its own state, and
the compiler helps us with that (the previous commits).
DEV-13163
This formalizes the previous commit a bit more and adds documentation
explaining why it exists and how it works. Look there for more
information.
This has been a lot of setup work. Hopefully things are now easier in the
future. And now we have nice declarative type-level hooks into the graph!
DEV-13163
This change is the first to utilize matching on edges to determine the state
of the template (to begin to derive its shape).
But this is notable for my finally caving on `min_specialization`.
The commit contains a bunch of rationale for why I introduced it. I've been
sitting on trying it for _years_. I had hoped for further progress in
determining a stabalization path, but that doesn't seem to be happening.
The reason I caved is because _not_ using it is a significant barrier to
utilizing robust types in various scenarios. I've been having to work
around that with significant efforts to write boilerplate code to match on
types and branch to various static paths accordingly. It makes it really
expensive to make certain types of changes, and it make the code really
difficult to understand once you start to peel back abstractions that try to
hide it.
I'll see how this goes and, if it goes well, begin to replace old methods
with specialization.
See the next commit for some cleanup. I purposefully left this a bit of a
mess (at the bottom of `asg::graph::object::tpl`) to emphasize what I'm
doing and why I introduced it.
DEV-13163
This allows for a declarative matching on edge targets using the trait
system, rather than having to convert the type to a runtime value to match
on (which doesn't make a whole lot of sense).
See a commit to follow shortly (with Tpl) for an example use case.
DEV-13163
Since we're statically invoking a particular ObjectKind's method, we already
know the source type. Let's pre-narrow it for their (my) convenience.
DEV-13163
There's a lot to say about this; it's been a bit of a struggle figuring out
what I wanted to do here.
First: this allows objects to use `AsgObjectMut` to control whether an edge
is permitted to be added, or to cache information about an edge that is
about to be added. But no object does that yet; it just uses the default
trait implementation, and so this _does not change any current
behavior_. It also is approximately equivalent cycle-count-wise, according
to Valgrind (within ~100 cycles out of hundreds of millions on large package
tests).
Adding edges to the graph is still infallible _after having received
permission_ from an `ObjectIndexRelTo`, but the object is free to reject the
edge with an `AsgError`.
As an example of where this will be useful: the template system needs to
keep track of what is in the body of a template as it is defined. But the
`TplAirAggregate` parser is sidelined while expressions in the body are
parsed, and edges are added to a dynamic source using
`ObjectIndexRelTo`. Consequently, we cannot rely on a static API to cache
information; we have to be able to react dynamically. This will allow `Tpl`
objects to know any time edges are added and, therefore, determine their
shape as the graph is being built, rather than having to traverse the tree
after encountering a close.
(I _could_ change this, but `ObjectIndexRelTo` removes a significant amount
of complexity for the caller, so I'd rather not.)
I did explore other options. I rejected the first one, then rejected this
one, then rejected the first one again before returning back to this one
after having previously sidelined the entire thing, because of the above
example. The core point is: I need confidence that the graph isn't being
changed in ways that I forgot about, and because of the complexity of the
system and the heavy refactoring that I do, I need the compiler's help;
otherwise I risk introducing subtle bugs as objects get out of sync with the
actual state of the graph.
(I wish the graph supported these things directly, but that's a project well
outside the scope of my TAMER work. So I have to make do, as I have been
all this time, by layering atop of Petgraph.)
(...I'm beginning to ramble.)
(...beginning?)
Anyway: my other rejected idea was to provide attestation via the
`ObjectIndex` APIs to force callers to go through those APIs to add an edge
to the graph; it would use sealed objects that are inaccessible to any
modules other than the objects, and assert that the caller is able to
provide a zero-sized object of that sealed type.
The problem with this is...exactly what was mentioned above:
`ObjectIndexRelTo` is dynamic. We don't always know the source object type
statically, and so we cannot make those static assertions.
I could have tried the same tricks to store attestation at some other time,
but what a confusing mess it would be.
And so here we are.
Most of this work is cleaning up the callers---adding edges is now fallible,
from the `ObjectIndex` API standpoint, and so AIR needed to be set up to
handle those failures. There _aren't_ any failures yet, but again, since
things are dynamic, they could appear at any moment. Furthermore, since
ref/def is commutative (things can be defined and referenced in any order),
there could be surprise errors on edge additions in places that might not
otherwise expect it in the future. We're now ready for that, and I'll be
able to e.g. traverse incoming edges on a `Missing->Transparent` definition
to notify dependents.
This project is going to be the end of me. As interesting as it is.
I can see why Rust just chose to require macro definitions _before_ use. So
much less work.
DEV-13163
AIR is no longer able to explicitly add edges without going through an
object-specific `ObjectIndex` API. `Asg::add_edge` was already private, but
`ObjectIndex::add_edge_{to,from}` was not.
The problem is that I want to augment the graph with other invariants, such
as caches. I'd normally have this built into the graph system itself, but I
don't have the time for the engineering effort to extend or replace
Petgraph, so I'm going to build atop of it.
To have confidence in any sort of caching, I need assurances that the graph
can't change out from underneath an object. This gets _close_ to
accomplishing that, but I'm still uncomfortable:
- We're one `pub` addition away from breaking these invariants; and
- Other `Object` types can still manipulates one-anothers' edges.
So this is a first step that at least proves encapsulation within
`asg::graph`, but ideally we'd have the system enforce, statically, that
`Objects` own their _outgoing_ edges, and no other `Object` is able to
manipulate them. This would ensure that any accidental future changes, or
bugs, will cause compilation failures rather than e.g. allowing caches to
get out of sync with the graph.
DEV-13163
The fixpoint tests for `meta-interp` are finally working. I could have
broken this up more, but I'm exhausted with this process, so, you get what
you get.
NIR will now recognize basic `<text>` and `<param-value>` nodes (note the
caveat for `<text>` in the comment, for now), and I finally include abstract
binding in the lowering pipeline. `xmli` output is also now able to cope
with metavariables with a single lexical association, and continues to
become more of a mess.
DEV-13163
This is a really obvious problem in retrospect, which makes me feel rather
silly.
The output was useful, but I don't have time to deal with this any further
right now. The comments in the commit explain the problem---that the output
ends up being interpolated as part of the fixpoint test, in an incorrect
context, and so the code that we generate is invalid. Also goes to show why
the fixpoint tests are important.
(Yes, they're still disabled for meta-interp, I'm trying to get them
enabled.)
DEV-13163
The provided documentation provides rationale, and the use case is the
ontree change. I was uncomfortable without the exhaustive match, and I was
further annoyed by the lack of easy `ObjectIndex` narrowing.
DEV-13163
This introduces the ability to specify an edge ordering for the ontological
tree traversal. `tree_reconstruction` will now use a
`SourceCompatibleTreeEdgeOrder`, which will traverse the graph in an order
that will result in a properly ordered source reconstruction. This is
needed for template headers, because interpolation causes
metavariables (exposed as template params) to be mixed into the body.
There's a lot of information here, including some TODOs on possible
improvements. I used the unstable `is_sorted` to output how many template
were already sorted, based on one of our very large packages internally that
uses templates extensively, and found that none of the desugared shorthand
template expansions were already ordered. If I tweak that a bit, then
nearly all templates will already be ordered, reducing the work that needs
to be done, leaving only template definitions with interpolation to be
concerned about, which is infrequent relative to everything else.
DEV-13163
Well, this is both good news and bad news.
The good news is that this finally produces the expected output and
reconstructs sources from interpolated values on the ASG. Yay!
...the bad news is that it's wrong. Notice how the fixpoint test is
disabled.
So, my plan was originally to commit it like this first and see if I was
comfortable relaxing the convention that `<param>` nodes had to appear in
the header. That's nice to do, that's cleaner to do, but would the
XSLT-based compiler really care? I had to investigate.
Well, turns out that TAMER does care. Because, well over a decade ago, I
re-used `<param>`, which could represent not only a template param, but also
a global param, or a function param.
So, XML->NIR considers all `<param>` nodes at the head of a template to be
template parameters. But after the first non-header element, we transition
to another state that allows it to be pretty much anything.
And so, I can't relax that restriction.
And because of that, I can't just stream the tree to the xmli generator,
I'll have to queue up nodes and order them.
Oh well, I tried.
DEV-13163
I'm not sure how I overlooked this previously, and I didn't notice until
trying to generate xmli output. I think I distracted myself with the
use of dangling status, which was not appropriate, and that has since
changed so that we have a dedicated concept.
This introduces the term "instantiation", or more specifically "lexical
instantiation". This is more specific and meaningful than simply
"expansion", which is what occurs during instantiation. I'll try to adjust
terminology and make things more consistent as I go.
DEV-13163
This logic ought to live alongside other definition logic...which in turn
needs its own extraction, but that's a separate concern.
This makes the definition of abstract identifiers very similar to
concrete. But, treating these as dangling, even if that's technically true,
has to change---we still want an edge drawn to the abstract identifier via
e.g. a template since we want the graph to mirror the structure of what it
will expand into concretely. I didn't notice this problem until trying to
generate the xmli for it.
So, see the commit to follow.
DEV-13163
This handles the common cases for meta, which includes what interpolation
desugars into. Most of this work was in testing and reasoning about the
issue; `asg::graph::visit:ontree::test` has a good summary of the structure
of the graph that results.
The last remaining steps to make this work end-to-end is for NIR->AIR to
lower `Nir::Ref` into `Air::BindIdent`, and then for `asg::graph::xmli` to
reconstruct concatenation lists. I'll then be able to commit the xmli test
case I've been sitting on, whose errors have been guiding my development.
DEV-13163
The term "metasyntactic" made sense literally---it's a variable in a
metalanguage that expands into a context that is able to contribute to the
language's syntax. But, the term has a different conventional use in
programming that is misleading.
The term "metalinguistic" is used in mathematics, to describe a metalanguage
or schema atop of a language. This is more fitting.
DEV-13163
This makes `SPair` construction more concise, getting rid of the `into`
invocations. For now I have only made this change in AIR's tests, since
that's what I'm working on and I want to observe how this convention
evolves. This may also encourage other changes, e.g. placing spans within
the `toks` array, rather than having to jump around the test for them.
The comment for `spair` mentions why this is a test-only function. But it
also shows how dangerous `impl Into<SymbolId> for &str` can be, since it
seems so innocuous---it uses a global interner. I'll be interested to see a
year from now if I decided to forego that impl in favor of explicit
internment, since I'm not sure it's worth the convenience anymore.
DEV-13163
This has been bothering me for quite a long time, and is just more test
cleanup before I introduce more. I suspect this came from habit with the
previous Rust edition where `into_iter()` on arrays was a much more verbose
operation.
To be clear: this change isn't for performance. It's about not doing
something silly when it's unnecessary, which also sets a bad example for
others.
There are many other tests in other modules that will need updating at some
point.
DEV-13163
This produces a representation of abstract identifiers on the graph, for
`Expr`s at least. The next step will probably be to get this working
end-to-end in the xmli output before extending it to the other remaining
bindable contexts.
DEV-13163
This enforces the naming convention that is utilized to infer whether an
identifier binding must be translated to an abstract binding.
This does not yet place any restrictions on other characters in identifier
names; both the placement of and flexibility of that has yet to be
decided. This change is sufficient enough to make abstract binding
translation reliable.
DEV-13163
The previous commit made me uncomfortable; we're already parsing with great
precision (and effort!) the grammar of NIR, and know for certain whether
we're in a metavariable binding context, so it makes no sense to have to try
to guess at another point in the lowering pipeline.
This introduces a new token to retain that information from XIR->NIR
lowering and then re-simplifies the lowering operation that was just
introduced in the previous commit (`AbstractBindTranslate`).
DEV-13163
This builds upon the concepts of the previous commit to translate identifier
binding into an abstract binding if it utilizes a symbol that follows a
metavariable naming convention.
See the provided documentation for more information.
This commit _does not_ integrate this into the lowering pipeline yet, since
the abstract identifiers are still rejected (as TODOs) by AIR.
DEV-13163
This introduces the notion of an abstract identifier, where the previous
identifiers are concrete. This serves as a compromise to either introducing
a new object type (another `Ident`), or having every `Ident` name be defined
by a `Meta` edge, which would bloat the graph significantly.
This change causes interpolation within a bind context to desugar into a new
`BindIdentAbstract` token, but AIR will throw an error if it encounters it
for now; that implementation will come soon.
This does not yet handle non-interpolation cases,
e.g. `<classify as="@foo@">`. This is a well-established shorthand for
`as="{@foo@}"`, but is unfortunately ambiguous in the context of
metavariable definitions (template parameters). This language ambiguity
will have to be handled here, and will have to fall back to today's behavior
of assuming concrete in that `param/@name` context but abstract every else,
unless of course interpolation is triggered using `{}` to disambiguate (as
in `<param name="{@foo@}"`).
I was going to handle the short-hand meta binding case as part of
interpolation, but I decided it may be appropriate for its own lowering
operation, since it is intended to work regardless of whether interpolation
takes place; it's a _translation_ of a binding into an abstract one, and it
can clearly delineate the awkward syntactic rules that we have to inherit,
as mentioned above.
DEV-13163
This prepares to make the name of an `Ident` optional to support abstract
identifiers derived from metavariables.
This is an unfortunate change to have to prepare for, since it complicates
how Idents are interpreted, but the alternative (a new object type) is not
good either. We'll see how this evolves.
DEV-13163
Mike Gerwitz