tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
// ASG IR
|
|
|
|
|
//
|
2023-01-17 23:09:25 -05:00
|
|
|
|
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
//
|
|
|
|
|
// This file is part of TAME.
|
|
|
|
|
//
|
|
|
|
|
// This program is free software: you can redistribute it and/or modify
|
|
|
|
|
// it under the terms of the GNU General Public License as published by
|
|
|
|
|
// the Free Software Foundation, either version 3 of the License, or
|
|
|
|
|
// (at your option) any later version.
|
|
|
|
|
//
|
|
|
|
|
// This program is distributed in the hope that it will be useful,
|
|
|
|
|
// but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
|
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
|
// GNU General Public License for more details.
|
|
|
|
|
//
|
|
|
|
|
// You should have received a copy of the GNU General Public License
|
|
|
|
|
// along with this program. If not, see <http://www.gnu.org/licenses/>.
|
|
|
|
|
|
tamer: asg::air: AIR as a sum IR
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
2023-03-02 15:15:28 -05:00
|
|
|
|
//! Intermediate representation for construction of the
|
|
|
|
|
//! [abstract semantic graph (ASG)](super) (AIR).
|
|
|
|
|
//!
|
|
|
|
|
//! AIR serves as an abstraction layer between higher-level parsers and the
|
|
|
|
|
//! aggregate ASG.
|
|
|
|
|
//! It allows parsers to operate as a raw stream of data without having to
|
|
|
|
|
//! worry about ownership of or references to the ASG,
|
|
|
|
|
//! and allows for multiple such parsers to be joined.
|
|
|
|
|
//!
|
|
|
|
|
//! AIR is _not_ intended to replace the API of the ASG---it
|
|
|
|
|
//! is intended as a termination point for the parsing pipeline,
|
|
|
|
|
//! and as such implements a subset of the ASG's API that is suitable
|
|
|
|
|
//! for aggregating raw data from source and object files.
|
|
|
|
|
//! Given that it does so little and is so close to the [`Asg`] API,
|
|
|
|
|
//! one might say that the abstraction is as light as air,
|
|
|
|
|
//! but that would surely result in face-palming and so we're not going
|
|
|
|
|
//! air such cringeworthy dad jokes here.
|
|
|
|
|
|
2023-03-28 16:14:09 -04:00
|
|
|
|
use super::{
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
graph::object::{Object, ObjectIndexTo, ObjectIndexToTree, Pkg, Root, Tpl},
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
Asg, AsgError, Expr, Ident, ObjectIndex,
|
2023-03-28 16:14:09 -04:00
|
|
|
|
};
|
2023-03-17 10:37:14 -04:00
|
|
|
|
use crate::{
|
2023-05-16 14:52:01 -04:00
|
|
|
|
f::Functor,
|
2023-03-30 15:10:52 -04:00
|
|
|
|
parse::{prelude::*, StateStack},
|
2023-05-10 11:16:07 -04:00
|
|
|
|
span::Span,
|
2023-05-04 15:03:14 -04:00
|
|
|
|
sym::SymbolId,
|
2023-03-17 10:37:14 -04:00
|
|
|
|
};
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
use std::fmt::{Debug, Display};
|
|
|
|
|
|
tamer: asg::air: AIR as a sum IR
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
2023-03-02 15:15:28 -05:00
|
|
|
|
#[macro_use]
|
|
|
|
|
mod ir;
|
|
|
|
|
pub use ir::Air;
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
|
2023-03-07 13:35:01 -05:00
|
|
|
|
mod expr;
|
2023-05-10 11:16:07 -04:00
|
|
|
|
mod pkg;
|
2023-03-07 14:00:46 -05:00
|
|
|
|
mod tpl;
|
2023-03-07 13:35:01 -05:00
|
|
|
|
use expr::AirExprAggregate;
|
2023-05-10 11:16:07 -04:00
|
|
|
|
use pkg::AirPkgAggregate;
|
2023-03-07 14:00:46 -05:00
|
|
|
|
use tpl::AirTplAggregate;
|
2023-03-07 13:35:01 -05:00
|
|
|
|
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
pub type IdentSym = SymbolId;
|
|
|
|
|
pub type DepSym = SymbolId;
|
|
|
|
|
|
tamer: asg::air::AirAggregate: Initial impl of nested exprs
This introduces a number of concepts together, again to demonstrate that
they were derived.
This introduces support for nested expressions, extending the previous
work. It also supports error recovery for dangling expressions.
The parser states are a mess; there is a lot of duplicate code here that
needs refactoring, but I wanted to commit this first at a known-good state
so that the diff will demonstrate the need for the change that will
follow; the opportunities for abstraction are plainly visible.
The immutable stack introduced here could be generalized, if needed, in the
future.
Another important note is that Rust optimizes away the `memcpy`s for the
stack that was introduced here. The initial Parser Context was introduced
because of `ArrayVec` inhibiting that elision, but Vec never had that
problem. In the future, I may choose to go back and remove ArrayVec, but I
had wanted to keep memory allocation out of the picture as much as possible
to make the disassembly and call graph easier to reason about and to have
confidence that optimizations were being performed as intended.
With that said---it _should_ be eliding in tamec, since we're not doing
anything meaningful yet with the graph. It does also elide in tameld, but
it's possible that Rust recognizes that those code paths are never taken
because tameld does nothing with expressions. So I'll have to monitor this
as I progress and adjust accordingly; it's possible a future commit will
call BS on everything I just said.
Of course, the counter-point to that is that Rust is optimizing them away
anyway, but Vec _does_ still require allocation; I was hoping to keep such
allocation at the fringes. But another counter-point is that it _still_ is
allocated at the fringe, when the context is initialized for the parser as
part of the lowering pipeline. But I didn't know how that would all come
together back then.
...alright, enough rambling.
DEV-13160
2023-01-05 15:57:06 -05:00
|
|
|
|
/// AIR parser state.
|
2023-03-07 16:28:32 -05:00
|
|
|
|
#[derive(Debug, PartialEq, Default)]
|
2022-06-02 13:26:46 -04:00
|
|
|
|
pub enum AirAggregate {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
/// Parser has not yet been initialized.
|
|
|
|
|
#[default]
|
|
|
|
|
Uninit,
|
|
|
|
|
|
tamer: asg::air: Begin lexical identifier resolution from bottom up
This begins demonstrating that the root will be utilized for identifier
lookup and indexing, as it was originally for TAME and is currently for the
linker.
This was _not_ the original plan---the plan was to have identifiers indexed
only at the package level, at least until we need a global lookup for
something else---but that plan was upended by how externs are currently
handled. So, for now, we need a global scope.
(Externs are resolved by the linker in such a way that _any_ package that
happens to be imported transitively may resolve the import. This is a
global environment, which I had hoped to get rid of, and which will need to
eventually go away (possibly along with externs) to support loading multiple
programs into the graph simultaneously for cross-program analysis.)
This commit renames the base state for `AirAggregate` to emphasize the fact,
especially when observing it in the `AirStack`, and changes
`AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of
the stack upward, rather than reverse, to prove that the system still
operates correctly with this change in place.
The reason for this direction change is to simplify lookup in the most
general case of non-local identifiers, which are almost all of them in
practice---they'll be immediately resolved at the root once they're
indexed. This can be done because I determined that I will _not_ support
shadowing; rationale for that will come later, but TAME is intended to be a
language suitable for non-programmer audiences as well. Note that
identifiers will be resolved lexically within templates in TAMER, unlike
TAME, which means that the expansion context will _not_ be considered when
checking for shadowing, so templates will still be able to compose without a
problem so long as they do not shadow in their definition context. (I'll
have to consider how that affects template-generating templates later on,
but that's an ambiguous construction in TAME today anyway.)
This _does not_ yet index anything at the root where it wasn't already being
indexed explicitly.
DEV-13162
2023-05-10 14:43:33 -04:00
|
|
|
|
/// Parser is in the root context.
|
|
|
|
|
///
|
|
|
|
|
/// As a parser,
|
|
|
|
|
/// this does nothing but await work.
|
|
|
|
|
/// Its presence in the [`AirStack`] is used for the global environment.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Root(ObjectIndex<Root>),
|
tamer: Initial concept for AIR/ASG Expr
This begins to place expressions on the graph---something that I've been
thinking about for a couple of years now, so it's interesting to finally be
doing it.
This is going to evolve; I want to get some things committed so that it's
clear how I'm moving forward. The ASG makes things a bit awkward for a
number of reasons:
1. I'm dealing with older code where I had a different model of doing
things;
2. It's mutable, rather than the mostly-functional lowering pipeline;
3. We're dealing with an aggregate ever-evolving blob of data (the graph)
rather than a stream of tokens; and
4. We don't have as many type guarantees.
I've shown with the lowering pipeline that I'm able to take a mutable
reference and convert it into something that's both functional and
performant, where I remove it from its container (an `Option`), create a new
version of it, and place it back. Rust is able to optimize away the memcpys
and such and just directly manipulate the underlying value, which is often a
register with all of the inlining.
_But_ this is a different scenario now. The lowering pipeline has a narrow
context. The graph has to keep hitting memory. So we'll see how this
goes. But it's most important to get this working and measure how it
performs; I'm not trying to prematurely optimize. My attempts right now are
for the way that I wish to develop.
Speaking to #4 above, it also sucks that I'm not able to type the
relationships between nodes on the graph. Rather, it's not that I _can't_,
but a project to created a typed graph library is beyond the scope of this
work and would take far too much time. I'll leave that to a personal,
non-work project. Instead, I'm going to have to narrow the type any time
the graph is accessed. And while that sucks, I'm going to do my best to
encapsulate those details to make it as seamless as possible API-wise. The
performance hit of performing the narrowing I'm hoping will be very small
relative to all the business logic going on (a single cache miss is bound to
be far more expensive than many narrowings which are just integer
comparisons and branching)...but we'll see. Introducing branching sucks,
but branch prediction is pretty damn good in modern CPUs.
DEV-13160
2022-12-21 16:47:04 -05:00
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
/// Parsing a package.
|
|
|
|
|
Pkg(AirPkgAggregate),
|
2023-01-30 16:51:24 -05:00
|
|
|
|
|
2023-03-06 12:03:55 -05:00
|
|
|
|
/// Parsing an expression.
|
tamer: Initial concept for AIR/ASG Expr
This begins to place expressions on the graph---something that I've been
thinking about for a couple of years now, so it's interesting to finally be
doing it.
This is going to evolve; I want to get some things committed so that it's
clear how I'm moving forward. The ASG makes things a bit awkward for a
number of reasons:
1. I'm dealing with older code where I had a different model of doing
things;
2. It's mutable, rather than the mostly-functional lowering pipeline;
3. We're dealing with an aggregate ever-evolving blob of data (the graph)
rather than a stream of tokens; and
4. We don't have as many type guarantees.
I've shown with the lowering pipeline that I'm able to take a mutable
reference and convert it into something that's both functional and
performant, where I remove it from its container (an `Option`), create a new
version of it, and place it back. Rust is able to optimize away the memcpys
and such and just directly manipulate the underlying value, which is often a
register with all of the inlining.
_But_ this is a different scenario now. The lowering pipeline has a narrow
context. The graph has to keep hitting memory. So we'll see how this
goes. But it's most important to get this working and measure how it
performs; I'm not trying to prematurely optimize. My attempts right now are
for the way that I wish to develop.
Speaking to #4 above, it also sucks that I'm not able to type the
relationships between nodes on the graph. Rather, it's not that I _can't_,
but a project to created a typed graph library is beyond the scope of this
work and would take far too much time. I'll leave that to a personal,
non-work project. Instead, I'm going to have to narrow the type any time
the graph is accessed. And while that sucks, I'm going to do my best to
encapsulate those details to make it as seamless as possible API-wise. The
performance hit of performing the narrowing I'm hoping will be very small
relative to all the business logic going on (a single cache miss is bound to
be far more expensive than many narrowings which are just integer
comparisons and branching)...but we'll see. Introducing branching sucks,
but branch prediction is pretty damn good in modern CPUs.
DEV-13160
2022-12-21 16:47:04 -05:00
|
|
|
|
///
|
2023-03-06 12:03:55 -05:00
|
|
|
|
/// This expects to inherit an [`AirExprAggregate`] from the prior state
|
|
|
|
|
/// so that we are not continuously re-allocating its stack for each
|
|
|
|
|
/// new expression root.
|
2023-03-29 13:49:05 -04:00
|
|
|
|
PkgExpr(AirExprAggregate),
|
2023-02-28 15:31:49 -05:00
|
|
|
|
|
|
|
|
|
/// Parser is in template parsing mode.
|
|
|
|
|
///
|
2023-03-15 10:59:22 -04:00
|
|
|
|
/// All objects encountered until the closing [`Air::TplEnd`] will be
|
2023-02-28 15:31:49 -05:00
|
|
|
|
/// parented to this template rather than the parent [`Pkg`].
|
2023-03-15 10:59:22 -04:00
|
|
|
|
/// See [`Air::TplStart`] for more information.
|
2023-03-29 09:33:40 -04:00
|
|
|
|
PkgTpl(AirTplAggregate),
|
tamer: asg::air::AirAggregate: Initial impl of nested exprs
This introduces a number of concepts together, again to demonstrate that
they were derived.
This introduces support for nested expressions, extending the previous
work. It also supports error recovery for dangling expressions.
The parser states are a mess; there is a lot of duplicate code here that
needs refactoring, but I wanted to commit this first at a known-good state
so that the diff will demonstrate the need for the change that will
follow; the opportunities for abstraction are plainly visible.
The immutable stack introduced here could be generalized, if needed, in the
future.
Another important note is that Rust optimizes away the `memcpy`s for the
stack that was introduced here. The initial Parser Context was introduced
because of `ArrayVec` inhibiting that elision, but Vec never had that
problem. In the future, I may choose to go back and remove ArrayVec, but I
had wanted to keep memory allocation out of the picture as much as possible
to make the disassembly and call graph easier to reason about and to have
confidence that optimizations were being performed as intended.
With that said---it _should_ be eliding in tamec, since we're not doing
anything meaningful yet with the graph. It does also elide in tameld, but
it's possible that Rust recognizes that those code paths are never taken
because tameld does nothing with expressions. So I'll have to monitor this
as I progress and adjust accordingly; it's possible a future commit will
call BS on everything I just said.
Of course, the counter-point to that is that Rust is optimizing them away
anyway, but Vec _does_ still require allocation; I was hoping to keep such
allocation at the fringes. But another counter-point is that it _still_ is
allocated at the fringe, when the context is initialized for the parser as
part of the lowering pipeline. But I didn't know how that would all come
together back then.
...alright, enough rambling.
DEV-13160
2023-01-05 15:57:06 -05:00
|
|
|
|
}
|
|
|
|
|
|
tamer: Initial concept for AIR/ASG Expr
This begins to place expressions on the graph---something that I've been
thinking about for a couple of years now, so it's interesting to finally be
doing it.
This is going to evolve; I want to get some things committed so that it's
clear how I'm moving forward. The ASG makes things a bit awkward for a
number of reasons:
1. I'm dealing with older code where I had a different model of doing
things;
2. It's mutable, rather than the mostly-functional lowering pipeline;
3. We're dealing with an aggregate ever-evolving blob of data (the graph)
rather than a stream of tokens; and
4. We don't have as many type guarantees.
I've shown with the lowering pipeline that I'm able to take a mutable
reference and convert it into something that's both functional and
performant, where I remove it from its container (an `Option`), create a new
version of it, and place it back. Rust is able to optimize away the memcpys
and such and just directly manipulate the underlying value, which is often a
register with all of the inlining.
_But_ this is a different scenario now. The lowering pipeline has a narrow
context. The graph has to keep hitting memory. So we'll see how this
goes. But it's most important to get this working and measure how it
performs; I'm not trying to prematurely optimize. My attempts right now are
for the way that I wish to develop.
Speaking to #4 above, it also sucks that I'm not able to type the
relationships between nodes on the graph. Rather, it's not that I _can't_,
but a project to created a typed graph library is beyond the scope of this
work and would take far too much time. I'll leave that to a personal,
non-work project. Instead, I'm going to have to narrow the type any time
the graph is accessed. And while that sucks, I'm going to do my best to
encapsulate those details to make it as seamless as possible API-wise. The
performance hit of performing the narrowing I'm hoping will be very small
relative to all the business logic going on (a single cache miss is bound to
be far more expensive than many narrowings which are just integer
comparisons and branching)...but we'll see. Introducing branching sucks,
but branch prediction is pretty damn good in modern CPUs.
DEV-13160
2022-12-21 16:47:04 -05:00
|
|
|
|
impl Display for AirAggregate {
|
|
|
|
|
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => write!(f, "awaiting AIR input"),
|
|
|
|
|
Root(_) => write!(f, "awaiting input at root"),
|
2023-05-10 11:16:07 -04:00
|
|
|
|
Pkg(pkg) => {
|
|
|
|
|
write!(f, "defining a package: {pkg}")
|
2023-03-06 12:03:55 -05:00
|
|
|
|
}
|
2023-03-29 09:33:40 -04:00
|
|
|
|
PkgExpr(expr) => {
|
2023-03-06 12:03:55 -05:00
|
|
|
|
write!(f, "defining a package expression: {expr}")
|
tamer: asg::air::AirAggregate: Initial impl of nested exprs
This introduces a number of concepts together, again to demonstrate that
they were derived.
This introduces support for nested expressions, extending the previous
work. It also supports error recovery for dangling expressions.
The parser states are a mess; there is a lot of duplicate code here that
needs refactoring, but I wanted to commit this first at a known-good state
so that the diff will demonstrate the need for the change that will
follow; the opportunities for abstraction are plainly visible.
The immutable stack introduced here could be generalized, if needed, in the
future.
Another important note is that Rust optimizes away the `memcpy`s for the
stack that was introduced here. The initial Parser Context was introduced
because of `ArrayVec` inhibiting that elision, but Vec never had that
problem. In the future, I may choose to go back and remove ArrayVec, but I
had wanted to keep memory allocation out of the picture as much as possible
to make the disassembly and call graph easier to reason about and to have
confidence that optimizations were being performed as intended.
With that said---it _should_ be eliding in tamec, since we're not doing
anything meaningful yet with the graph. It does also elide in tameld, but
it's possible that Rust recognizes that those code paths are never taken
because tameld does nothing with expressions. So I'll have to monitor this
as I progress and adjust accordingly; it's possible a future commit will
call BS on everything I just said.
Of course, the counter-point to that is that Rust is optimizing them away
anyway, but Vec _does_ still require allocation; I was hoping to keep such
allocation at the fringes. But another counter-point is that it _still_ is
allocated at the fringe, when the context is initialized for the parser as
part of the lowering pipeline. But I didn't know how that would all come
together back then.
...alright, enough rambling.
DEV-13160
2023-01-05 15:57:06 -05:00
|
|
|
|
}
|
2023-03-29 09:33:40 -04:00
|
|
|
|
PkgTpl(tpl) => {
|
2023-03-07 12:41:47 -05:00
|
|
|
|
write!(f, "building a template: {tpl}",)
|
2023-02-28 15:31:49 -05:00
|
|
|
|
}
|
tamer: Initial concept for AIR/ASG Expr
This begins to place expressions on the graph---something that I've been
thinking about for a couple of years now, so it's interesting to finally be
doing it.
This is going to evolve; I want to get some things committed so that it's
clear how I'm moving forward. The ASG makes things a bit awkward for a
number of reasons:
1. I'm dealing with older code where I had a different model of doing
things;
2. It's mutable, rather than the mostly-functional lowering pipeline;
3. We're dealing with an aggregate ever-evolving blob of data (the graph)
rather than a stream of tokens; and
4. We don't have as many type guarantees.
I've shown with the lowering pipeline that I'm able to take a mutable
reference and convert it into something that's both functional and
performant, where I remove it from its container (an `Option`), create a new
version of it, and place it back. Rust is able to optimize away the memcpys
and such and just directly manipulate the underlying value, which is often a
register with all of the inlining.
_But_ this is a different scenario now. The lowering pipeline has a narrow
context. The graph has to keep hitting memory. So we'll see how this
goes. But it's most important to get this working and measure how it
performs; I'm not trying to prematurely optimize. My attempts right now are
for the way that I wish to develop.
Speaking to #4 above, it also sucks that I'm not able to type the
relationships between nodes on the graph. Rather, it's not that I _can't_,
but a project to created a typed graph library is beyond the scope of this
work and would take far too much time. I'll leave that to a personal,
non-work project. Instead, I'm going to have to narrow the type any time
the graph is accessed. And while that sucks, I'm going to do my best to
encapsulate those details to make it as seamless as possible API-wise. The
performance hit of performing the narrowing I'm hoping will be very small
relative to all the business logic going on (a single cache miss is bound to
be far more expensive than many narrowings which are just integer
comparisons and branching)...but we'll see. Introducing branching sucks,
but branch prediction is pretty damn good in modern CPUs.
DEV-13160
2022-12-21 16:47:04 -05:00
|
|
|
|
}
|
|
|
|
|
}
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
}
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
impl From<AirPkgAggregate> for AirAggregate {
|
|
|
|
|
fn from(st: AirPkgAggregate) -> Self {
|
|
|
|
|
Self::Pkg(st)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-29 13:49:05 -04:00
|
|
|
|
impl From<AirExprAggregate> for AirAggregate {
|
|
|
|
|
fn from(st: AirExprAggregate) -> Self {
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
Self::PkgExpr(st)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<AirTplAggregate> for AirAggregate {
|
|
|
|
|
fn from(st: AirTplAggregate) -> Self {
|
|
|
|
|
Self::PkgTpl(st)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2022-06-02 13:26:46 -04:00
|
|
|
|
impl ParseState for AirAggregate {
|
2022-12-13 14:36:38 -05:00
|
|
|
|
type Token = Air;
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
type Object = ();
|
|
|
|
|
type Error = AsgError;
|
tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
2023-03-27 11:47:11 -04:00
|
|
|
|
type Context = AirAggregateCtx;
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
|
|
|
|
|
/// Destination [`Asg`] that this parser lowers into.
|
|
|
|
|
///
|
tamer: asg::air: AIR as a sum IR
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
2023-03-02 15:15:28 -05:00
|
|
|
|
/// This ASG will be yielded by [`crate::parse::Parser::finalize`].
|
tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
2023-03-27 11:47:11 -04:00
|
|
|
|
type PubContext = Asg;
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
|
|
|
|
|
fn parse_token(
|
|
|
|
|
self,
|
|
|
|
|
tok: Self::Token,
|
tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
2023-03-27 11:47:11 -04:00
|
|
|
|
ctx: &mut Self::Context,
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
) -> crate::parse::TransitionResult<Self> {
|
2023-05-10 12:42:57 -04:00
|
|
|
|
use ir::{AirSubsets::*, AirTodo::*};
|
2022-06-02 13:26:46 -04:00
|
|
|
|
use AirAggregate::*;
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
|
tamer: asg::air: AIR as a sum IR
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
2023-03-02 15:15:28 -05:00
|
|
|
|
match (self, tok.into()) {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
// Initialize the parser with the graph root.
|
|
|
|
|
// The graph may contain multiple roots in the future to support
|
|
|
|
|
// cross-version analysis.
|
|
|
|
|
(Uninit, tok) => Transition(Root(ctx.asg_mut().root(tok.span())))
|
|
|
|
|
.incomplete()
|
|
|
|
|
.with_lookahead(tok),
|
|
|
|
|
|
tamer: asg::air: AIR as a sum IR
This introduces a new macro `sum_ir!` to help with a long-standing problem
of not being able to easily narrow types in Rust without a whole lot of
boilerplate. This patch includes a bit of documentation, so see that for
more information.
This was not a welcome change---I jumped down this rabbit hole trying to
decompose `AirAggregate` so that I can share portions of parsing with the
current parser and a template parser. I can now proceed with that.
This is not the only implementation that I had tried. I previously inverted
the approach, as I've been doing manually for some time: manually create
types to hold the sets of variants, and then create a sum type to hold those
types. That works, but it resulted in a mess for systems that have to use
the IR, since now you have two enums to contend with. I didn't find that to
be appropriate, because we shouldn't complicate the external API for
implementation details.
The enum for IRs is supposed to be like a bytecode---a list of operations
that can be performed with the IR. They can be grouped if it makes sense
for a public API, but in my case, I only wanted subsets for the sake of
delegating responsibilities to smaller subsystems, while retaining the
context that `match` provides via its exhaustiveness checking but does not
expose as something concrete (which is deeply frustrating!).
Anyway, here we are; this'll be refined over time, hopefully, and
portions of it can be generalized for removing boilerplate from other IRs.
Another thing to note is that this syntax is really a compromise---I had to
move on, and I was spending too much time trying to get creative with
`macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in
some places and is therefore not necessarily all that intuitive. This can
be refined further in the future. But the end result, all things
considered, isn't too bad.
DEV-13708
2023-03-02 15:15:28 -05:00
|
|
|
|
(st, AirTodo(Todo(_))) => Transition(st).incomplete(),
|
tamer: asg::air::AirAggregate: Initial impl of nested exprs
This introduces a number of concepts together, again to demonstrate that
they were derived.
This introduces support for nested expressions, extending the previous
work. It also supports error recovery for dangling expressions.
The parser states are a mess; there is a lot of duplicate code here that
needs refactoring, but I wanted to commit this first at a known-good state
so that the diff will demonstrate the need for the change that will
follow; the opportunities for abstraction are plainly visible.
The immutable stack introduced here could be generalized, if needed, in the
future.
Another important note is that Rust optimizes away the `memcpy`s for the
stack that was introduced here. The initial Parser Context was introduced
because of `ArrayVec` inhibiting that elision, but Vec never had that
problem. In the future, I may choose to go back and remove ArrayVec, but I
had wanted to keep memory allocation out of the picture as much as possible
to make the disassembly and call graph easier to reason about and to have
confidence that optimizations were being performed as intended.
With that said---it _should_ be eliding in tamec, since we're not doing
anything meaningful yet with the graph. It does also elide in tameld, but
it's possible that Rust recognizes that those code paths are never taken
because tameld does nothing with expressions. So I'll have to monitor this
as I progress and adjust accordingly; it's possible a future commit will
call BS on everything I just said.
Of course, the counter-point to that is that Rust is optimizing them away
anyway, but Vec _does_ still require allocation; I was hoping to keep such
allocation at the fringes. But another counter-point is that it _still_ is
allocated at the fringe, when the context is initialized for the parser as
part of the lowering pipeline. But I didn't know how that would all come
together back then.
...alright, enough rambling.
DEV-13160
2023-01-05 15:57:06 -05:00
|
|
|
|
|
2023-05-10 12:42:57 -04:00
|
|
|
|
// Package
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
(st @ (Root(..) | PkgExpr(..) | PkgTpl(..)), tok @ AirPkg(..)) => {
|
2023-05-10 11:16:07 -04:00
|
|
|
|
ctx.ret_or_transfer(st, tok, AirPkgAggregate::new())
|
2023-04-12 10:03:37 -04:00
|
|
|
|
}
|
2023-05-10 11:16:07 -04:00
|
|
|
|
(Pkg(pkg), AirPkg(etok)) => ctx.proxy(pkg, etok),
|
|
|
|
|
(Pkg(pkg), AirBind(etok)) => ctx.proxy(pkg, etok),
|
|
|
|
|
(Pkg(pkg), AirIdent(etok)) => ctx.proxy(pkg, etok),
|
|
|
|
|
(Pkg(pkg), AirDoc(etok)) => ctx.proxy(pkg, etok),
|
2023-04-12 10:03:37 -04:00
|
|
|
|
|
2023-05-10 12:42:57 -04:00
|
|
|
|
// Expression
|
2023-05-10 11:16:07 -04:00
|
|
|
|
(st @ (Pkg(_) | PkgTpl(_)), tok @ AirExpr(..)) => {
|
2023-03-30 16:30:11 -04:00
|
|
|
|
ctx.ret_or_transfer(st, tok, AirExprAggregate::new())
|
2023-03-06 12:03:55 -05:00
|
|
|
|
}
|
2023-03-30 16:30:11 -04:00
|
|
|
|
(PkgExpr(expr), AirExpr(etok)) => ctx.proxy(expr, etok),
|
|
|
|
|
(PkgExpr(expr), AirBind(etok)) => ctx.proxy(expr, etok),
|
2023-04-10 12:16:21 -04:00
|
|
|
|
(PkgExpr(expr), AirDoc(etok)) => ctx.proxy(expr, etok),
|
2023-03-06 12:03:55 -05:00
|
|
|
|
|
2023-05-10 12:42:57 -04:00
|
|
|
|
// Template
|
2023-05-10 11:16:07 -04:00
|
|
|
|
(st @ (Pkg(_) | PkgExpr(_)), tok @ AirTpl(..)) => {
|
2023-03-30 16:30:11 -04:00
|
|
|
|
ctx.ret_or_transfer(st, tok, AirTplAggregate::new())
|
2023-03-11 00:40:54 -05:00
|
|
|
|
}
|
2023-03-30 16:30:11 -04:00
|
|
|
|
(PkgTpl(tplst), AirTpl(ttok)) => ctx.proxy(tplst, ttok),
|
|
|
|
|
(PkgTpl(tplst), AirBind(ttok)) => ctx.proxy(tplst, ttok),
|
2023-04-10 12:16:21 -04:00
|
|
|
|
(PkgTpl(tplst), AirDoc(ttok)) => ctx.proxy(tplst, ttok),
|
2023-03-11 00:40:54 -05:00
|
|
|
|
|
2023-04-10 12:16:21 -04:00
|
|
|
|
(
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
st @ Root(_),
|
2023-04-10 12:16:21 -04:00
|
|
|
|
tok @ (AirExpr(..) | AirBind(..) | AirTpl(..) | AirDoc(..)),
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
) => Transition(st).err(AsgError::PkgExpected(tok.span())),
|
2023-03-06 12:03:55 -05:00
|
|
|
|
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
(st @ (Root(..) | PkgExpr(..) | PkgTpl(..)), AirIdent(tok)) => {
|
2023-05-08 15:50:44 -04:00
|
|
|
|
Transition(st).err(AsgError::UnexpectedOpaqueIdent(tok.name()))
|
|
|
|
|
}
|
2023-03-06 12:03:55 -05:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
fn is_accepting(&self, ctx: &Self::Context) -> bool {
|
|
|
|
|
ctx.stack_ref().iter().all(|st| st.active_is_accepting(ctx))
|
|
|
|
|
&& self.active_is_accepting(ctx)
|
2023-03-06 12:03:55 -05:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl AirAggregate {
|
2023-05-10 11:16:07 -04:00
|
|
|
|
/// Whether the active parser is completed with active parsing.
|
|
|
|
|
///
|
|
|
|
|
/// This method is used to determine whether control ought to be
|
|
|
|
|
/// transferred to a new child parser.
|
2023-03-30 09:20:34 -04:00
|
|
|
|
///
|
|
|
|
|
/// If a child parser is active,
|
|
|
|
|
/// then its [`ParseState::is_accepting`] will be consulted.
|
2023-05-10 11:16:07 -04:00
|
|
|
|
fn active_is_complete(&self, ctx: &<Self as ParseState>::Context) -> bool {
|
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => false,
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
// We can't be done with something we're not doing.
|
|
|
|
|
// This is necessary to start the first child parser.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Root(_) => false,
|
2023-05-10 11:16:07 -04:00
|
|
|
|
|
|
|
|
|
Pkg(st) => st.is_accepting(ctx),
|
|
|
|
|
PkgExpr(st) => st.is_accepting(ctx),
|
|
|
|
|
PkgTpl(st) => st.is_accepting(ctx),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
// Whether the parser is in an accepting state.
|
2023-03-30 09:20:34 -04:00
|
|
|
|
fn active_is_accepting(&self, ctx: &<Self as ParseState>::Context) -> bool {
|
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => false,
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
// This must not recurse on `AirAggregate::is_accepting`,
|
|
|
|
|
// otherwise it'll be mutually recursive.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Root(_) => true,
|
2023-05-10 11:16:07 -04:00
|
|
|
|
|
|
|
|
|
Pkg(st) => st.is_accepting(ctx),
|
2023-03-30 09:20:34 -04:00
|
|
|
|
PkgExpr(st) => st.is_accepting(ctx),
|
|
|
|
|
PkgTpl(st) => st.is_accepting(ctx),
|
|
|
|
|
}
|
2023-03-07 12:41:47 -05:00
|
|
|
|
}
|
2023-04-05 10:45:54 -04:00
|
|
|
|
|
|
|
|
|
/// The rooting context for [`Ident`]s for the active parser.
|
|
|
|
|
///
|
|
|
|
|
/// A value of [`None`] indicates that the current parser does not
|
|
|
|
|
/// support direct bindings,
|
|
|
|
|
/// but a parent context may
|
|
|
|
|
/// (see [`AirAggregateCtx::rooting_oi`]).
|
|
|
|
|
fn active_rooting_oi(&self) -> Option<ObjectIndexToTree<Ident>> {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
|
2023-04-05 10:45:54 -04:00
|
|
|
|
match self {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => None,
|
|
|
|
|
|
|
|
|
|
// Root will serve as a pool of identifiers,
|
|
|
|
|
// but it can never _contain_ their definitions.
|
|
|
|
|
// See `active_env_oi`.
|
|
|
|
|
Root(_) => None,
|
2023-05-10 11:16:07 -04:00
|
|
|
|
|
|
|
|
|
// Packages always serve as roots for identifiers
|
|
|
|
|
// (that is their entire purpose).
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Pkg(pkgst) => pkgst.active_pkg_oi().map(Into::into),
|
2023-04-05 10:45:54 -04:00
|
|
|
|
|
|
|
|
|
// Expressions never serve as roots for identifiers;
|
|
|
|
|
// this will always fall through to the parent context.
|
|
|
|
|
// Since the parent context is a package or a template,
|
|
|
|
|
// the next frame should succeed.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
PkgExpr(_) => None,
|
2023-04-05 10:45:54 -04:00
|
|
|
|
|
|
|
|
|
// Identifiers bound while within a template definition context
|
|
|
|
|
// must bind to the eventual _expansion_ site,
|
|
|
|
|
// as if the body were pasted there.
|
|
|
|
|
// Templates must therefore serve as containers for identifiers
|
|
|
|
|
// bound therein.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Active environment for identifier lookups.
|
|
|
|
|
///
|
|
|
|
|
/// An environment is a superset of a container,
|
|
|
|
|
/// which is described by [`Self::active_rooting_oi`].
|
|
|
|
|
/// For example,
|
|
|
|
|
/// [`Self::Root`] cannot own any identifiers,
|
|
|
|
|
/// but it can serve as a pool of references to them.
|
|
|
|
|
fn active_env_oi(&self) -> Option<ObjectIndexTo<Ident>> {
|
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
|
|
|
|
Root(oi_root) => Some((*oi_root).into()),
|
|
|
|
|
_ => self.active_rooting_oi().map(Into::into),
|
2023-04-05 10:45:54 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
2023-05-16 14:52:01 -04:00
|
|
|
|
|
|
|
|
|
/// Adjust a [`EnvScopeKind`] while crossing an environment boundary
|
|
|
|
|
/// into `self`.
|
|
|
|
|
///
|
|
|
|
|
/// An identifier is _visible_ at the environment in which it is defined.
|
|
|
|
|
/// This identifier casts a _shadow_ to lower environments,
|
|
|
|
|
/// with the exception of the root.
|
|
|
|
|
/// The _root_ will absorb adjacent visible identifiers into a _pool_,
|
|
|
|
|
/// which is distinct from the hierarchy that is otherwise created at
|
|
|
|
|
/// the package level and lower.
|
|
|
|
|
fn env_cross_boundary_into<T>(
|
|
|
|
|
&self,
|
|
|
|
|
kind: EnvScopeKind<T>,
|
|
|
|
|
) -> EnvScopeKind<T> {
|
|
|
|
|
use AirAggregate::*;
|
|
|
|
|
use EnvScopeKind::*;
|
|
|
|
|
|
|
|
|
|
match (self, kind) {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
// This is not an environment.
|
|
|
|
|
(Uninit, kind) => kind,
|
|
|
|
|
|
2023-05-16 15:03:54 -04:00
|
|
|
|
// Hidden is a fixpoint.
|
|
|
|
|
(_, kind @ Hidden(_)) => kind,
|
2023-05-16 14:52:01 -04:00
|
|
|
|
|
|
|
|
|
// Expressions do not introduce their own environment
|
|
|
|
|
// (they are not containers)
|
|
|
|
|
// and so act as an identity function.
|
|
|
|
|
(PkgExpr(_), kind) => kind,
|
|
|
|
|
|
|
|
|
|
// A visible identifier will always cast a shadow in one step.
|
|
|
|
|
// A shadow will always be cast (propagate) until the root.
|
|
|
|
|
(Pkg(_) | PkgTpl(_), Visible(x) | Shadow(x)) => Shadow(x),
|
|
|
|
|
|
|
|
|
|
// Above we see that Visual will always transition to Shadow in
|
|
|
|
|
// one step.
|
|
|
|
|
// Consequently,
|
|
|
|
|
// Visible at Root means that we're a package-level Visible,
|
|
|
|
|
// which must contribute to the pool.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
(Root(_), Visible(x)) => Visible(x),
|
2023-05-16 14:52:01 -04:00
|
|
|
|
|
|
|
|
|
// If we're _not_ Visible at the root,
|
|
|
|
|
// then we're _not_ a package-level definition,
|
|
|
|
|
// and so we should _not_ contribute to the pool.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
(Root(_), Shadow(x)) => Hidden(x),
|
2023-05-16 14:52:01 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
2023-03-06 12:03:55 -05:00
|
|
|
|
}
|
|
|
|
|
|
2023-03-29 09:46:17 -04:00
|
|
|
|
/// Additional parser context,
|
|
|
|
|
/// including the ASG and parser stack frames.
|
2023-03-28 16:14:09 -04:00
|
|
|
|
///
|
2023-03-29 09:33:40 -04:00
|
|
|
|
/// [`ObjectIndex`] lookups perform reverse linear searches beginning from
|
|
|
|
|
/// the last stack frame until a non-[`None`] value is found;
|
|
|
|
|
/// this creates an environment whereby inner contexts shadow outer.
|
|
|
|
|
/// Missing values create holes,
|
|
|
|
|
/// much like a prototype chain.
|
|
|
|
|
/// In practice,
|
|
|
|
|
/// this should only have to search the last two frames.
|
2023-03-28 16:14:09 -04:00
|
|
|
|
#[derive(Debug, Default)]
|
2023-03-30 22:28:22 -04:00
|
|
|
|
pub struct AirAggregateCtx(Asg, AirStack, Option<ObjectIndex<Pkg>>);
|
2023-03-29 09:46:17 -04:00
|
|
|
|
|
2023-03-30 15:10:52 -04:00
|
|
|
|
/// Limit of the maximum number of held parser frames.
|
|
|
|
|
///
|
|
|
|
|
/// Note that this is the number of [`ParseState`]s held,
|
|
|
|
|
/// _not_ the depth of the graph at a given point.
|
|
|
|
|
/// The intent of this is to limit runaway recursion in the event of some
|
|
|
|
|
/// bug in the system;
|
|
|
|
|
/// while the input stream is certainly finite,
|
|
|
|
|
/// lookahead tokens cause recursion that does not provably
|
|
|
|
|
/// terminate.
|
|
|
|
|
///
|
|
|
|
|
/// This limit is arbitrarily large,
|
|
|
|
|
/// but hopefully such that no legitimate case will ever hit it.
|
|
|
|
|
const MAX_AIR_STACK_DEPTH: usize = 1024;
|
|
|
|
|
|
2023-03-29 09:46:17 -04:00
|
|
|
|
/// Held parser stack frames.
|
|
|
|
|
///
|
|
|
|
|
/// See [`AirAggregateCtx`] for more information.
|
2023-03-30 15:10:52 -04:00
|
|
|
|
pub type AirStack = StateStack<AirAggregate, MAX_AIR_STACK_DEPTH>;
|
2023-03-29 09:46:17 -04:00
|
|
|
|
|
|
|
|
|
impl AirAggregateCtx {
|
|
|
|
|
fn asg_mut(&mut self) -> &mut Asg {
|
|
|
|
|
self.as_mut()
|
|
|
|
|
}
|
2023-03-28 16:14:09 -04:00
|
|
|
|
|
2023-05-10 12:42:57 -04:00
|
|
|
|
fn asg_ref(&self) -> &Asg {
|
|
|
|
|
self.as_ref()
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-30 15:10:52 -04:00
|
|
|
|
fn stack(&mut self) -> &mut AirStack {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
let Self(_, stack, _) = self;
|
2023-03-30 15:10:52 -04:00
|
|
|
|
stack
|
2023-03-28 16:14:09 -04:00
|
|
|
|
}
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
fn stack_ref(&self) -> &AirStack {
|
|
|
|
|
let Self(_, stack, _) = self;
|
|
|
|
|
stack
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-30 16:30:11 -04:00
|
|
|
|
/// Return control to the parser atop of the stack if `st` is an
|
|
|
|
|
/// accepting state,
|
|
|
|
|
/// otherwise transfer control to a new parser `to`.
|
|
|
|
|
///
|
|
|
|
|
/// This serves as a balance with the behavior of [`Self::proxy`].
|
|
|
|
|
/// Rather than checking for an accepting state after each proxy,
|
|
|
|
|
/// or having the child parsers return to the top stack frame once
|
|
|
|
|
/// they have completed,
|
|
|
|
|
/// we leave the child parser in place to potentially handle more
|
|
|
|
|
/// tokens of the same type.
|
|
|
|
|
/// For example,
|
|
|
|
|
/// adjacent expressions can re-use the same parser rather than having
|
|
|
|
|
/// to pop and push for each sibling.
|
|
|
|
|
///
|
|
|
|
|
/// Consequently,
|
|
|
|
|
/// this means that a parser may be complete when we need to push and
|
|
|
|
|
/// transfer control to another parser.
|
|
|
|
|
/// Before pushing,
|
|
|
|
|
/// we first check to see if the parser atop of the stack is in an
|
|
|
|
|
/// accepting state.
|
|
|
|
|
/// If so,
|
|
|
|
|
/// then we are a sibling,
|
|
|
|
|
/// and so instead of proceeding with instantiating a new parser,
|
|
|
|
|
/// we return to the one atop of the stack and delegate to it.
|
|
|
|
|
///
|
|
|
|
|
/// If `st` is _not_ in an accepting state,
|
|
|
|
|
/// that means that we are a _child_;
|
|
|
|
|
/// we then set aside the state `st` on the stack and transfer
|
|
|
|
|
/// control to the child `to`.
|
|
|
|
|
///
|
|
|
|
|
/// See also [`Self::proxy`].
|
2023-04-05 10:45:54 -04:00
|
|
|
|
fn ret_or_transfer<S: Into<AirAggregate>, SB: Into<AirAggregate>>(
|
2023-03-30 16:30:11 -04:00
|
|
|
|
&mut self,
|
2023-04-05 10:45:54 -04:00
|
|
|
|
st: S,
|
2023-03-30 16:30:11 -04:00
|
|
|
|
tok: impl Token + Into<Air>,
|
2023-04-05 10:45:54 -04:00
|
|
|
|
to: SB,
|
2023-03-30 16:30:11 -04:00
|
|
|
|
) -> TransitionResult<AirAggregate> {
|
2023-04-05 10:45:54 -04:00
|
|
|
|
let st_super = st.into();
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
if st_super.active_is_complete(self) {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
// TODO: error (this should never happen, so maybe panic instead?)
|
|
|
|
|
self.stack().ret_or_dead(AirAggregate::Uninit, tok)
|
2023-03-30 16:30:11 -04:00
|
|
|
|
} else {
|
|
|
|
|
self.stack().transfer_with_ret(
|
2023-04-05 10:45:54 -04:00
|
|
|
|
Transition(st_super),
|
2023-03-30 16:30:11 -04:00
|
|
|
|
Transition(to.into()).incomplete().with_lookahead(tok),
|
|
|
|
|
)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Proxy `tok` to `st`,
|
|
|
|
|
/// returning to the state atop of the stack if parsing reaches a dead
|
|
|
|
|
/// state.
|
|
|
|
|
///
|
|
|
|
|
/// See also [`Self::ret_or_transfer`].
|
|
|
|
|
fn proxy<S: ParseState<Super = AirAggregate, Context = Self>>(
|
|
|
|
|
&mut self,
|
|
|
|
|
st: S,
|
|
|
|
|
tok: impl Token + Into<S::Token>,
|
|
|
|
|
) -> TransitionResult<AirAggregate> {
|
|
|
|
|
st.delegate_child(tok.into(), self, |_deadst, tok, ctx| {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
// TODO: error (this should never happen, so maybe panic instead?)
|
|
|
|
|
ctx.stack().ret_or_dead(AirAggregate::Uninit, tok)
|
2023-03-30 16:30:11 -04:00
|
|
|
|
})
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-30 22:28:22 -04:00
|
|
|
|
/// Create a new rooted package and record it as the active package.
|
2023-05-10 11:16:07 -04:00
|
|
|
|
fn pkg_begin(
|
2023-05-02 22:12:37 -04:00
|
|
|
|
&mut self,
|
|
|
|
|
start: Span,
|
|
|
|
|
name: SPair,
|
|
|
|
|
) -> Result<ObjectIndex<Pkg>, AsgError> {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
let Self(asg, _, pkg) = self;
|
2023-05-02 22:12:37 -04:00
|
|
|
|
|
|
|
|
|
let oi_root = asg.root(start);
|
|
|
|
|
let oi_pkg = oi_root.create_pkg(asg, start, name)?;
|
2023-03-30 22:28:22 -04:00
|
|
|
|
|
|
|
|
|
pkg.replace(oi_pkg);
|
2023-05-02 22:12:37 -04:00
|
|
|
|
Ok(oi_pkg)
|
2023-03-30 22:28:22 -04:00
|
|
|
|
}
|
|
|
|
|
|
2023-05-10 11:16:07 -04:00
|
|
|
|
/// Indicate that there is no longer any active package.
|
|
|
|
|
fn pkg_clear(&mut self) {
|
|
|
|
|
let Self(_, _, pkg) = self;
|
|
|
|
|
pkg.take();
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-30 22:28:22 -04:00
|
|
|
|
/// The active package if any.
|
|
|
|
|
fn pkg_oi(&self) -> Option<ObjectIndex<Pkg>> {
|
|
|
|
|
match self {
|
|
|
|
|
Self(_, _, oi) => *oi,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-31 13:57:11 -04:00
|
|
|
|
/// The active container (rooting context) for [`Ident`]s.
|
|
|
|
|
///
|
|
|
|
|
/// The integer value returned represents the stack offset at which the
|
|
|
|
|
/// rooting index was found,
|
|
|
|
|
/// with `0` representing the package.
|
2023-03-28 16:14:09 -04:00
|
|
|
|
///
|
|
|
|
|
/// A value of [`None`] indicates that no bindings are permitted in the
|
|
|
|
|
/// current context.
|
2023-04-05 10:45:54 -04:00
|
|
|
|
fn rooting_oi(&self) -> Option<ObjectIndexToTree<Ident>> {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
let Self(_, stack, _) = self;
|
2023-03-28 16:14:09 -04:00
|
|
|
|
|
2023-04-05 10:45:54 -04:00
|
|
|
|
stack.iter().rev().find_map(|st| st.active_rooting_oi())
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// The active dangling expression context for [`Expr`]s.
|
|
|
|
|
///
|
|
|
|
|
/// A value of [`None`] indicates that expressions are not permitted to
|
|
|
|
|
/// dangle in the current context
|
|
|
|
|
/// (and so must be identified).
|
|
|
|
|
fn dangling_expr_oi(&self) -> Option<ObjectIndexTo<Expr>> {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
use AirAggregate::*;
|
2023-03-30 22:28:22 -04:00
|
|
|
|
let Self(_, stack, _) = self;
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
|
|
|
|
|
stack.iter().rev().find_map(|st| match st {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => None,
|
|
|
|
|
|
|
|
|
|
// It should never be possible to define expressions directly in
|
|
|
|
|
// Root.
|
|
|
|
|
Root(_) => None,
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
|
|
|
|
|
// A dangling expression in a package context would be
|
|
|
|
|
// unreachable.
|
|
|
|
|
// There should be no parent frame and so this will fail to find
|
|
|
|
|
// a value.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Pkg(_) => None,
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
|
|
|
|
|
// Expressions may always contain other expressions,
|
|
|
|
|
// and so this method should not be consulted in such a
|
|
|
|
|
// context.
|
|
|
|
|
// Nonetheless,
|
|
|
|
|
// fall through to the parent frame and give a correct answer.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
PkgExpr(_) => None,
|
tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.
This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.
I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out. These commits serve as a proof of
necessity. Or, at least, concrete rationale.
It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser. This means that we're just about ready
for it to serve as a superstate. But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.
DEV-13708
2023-03-29 11:19:59 -04:00
|
|
|
|
|
|
|
|
|
// Templates serve as containers for dangling expressions,
|
|
|
|
|
// since they may expand into an context where they are not
|
|
|
|
|
// considered to be dangling.
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
|
2023-03-29 09:33:40 -04:00
|
|
|
|
})
|
2023-03-28 16:14:09 -04:00
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// The active expansion target (splicing context) for [`Tpl`]s.
|
|
|
|
|
///
|
|
|
|
|
/// A value of [`None`] indicates that template expansion is not
|
|
|
|
|
/// permitted in this current context.
|
|
|
|
|
fn expansion_oi(&self) -> Option<ObjectIndexTo<Tpl>> {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
use AirAggregate::*;
|
2023-03-30 22:28:22 -04:00
|
|
|
|
let Self(_, stack, _) = self;
|
2023-03-28 16:14:09 -04:00
|
|
|
|
|
2023-04-05 10:45:54 -04:00
|
|
|
|
stack.iter().rev().find_map(|st| match st {
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
Uninit => None,
|
|
|
|
|
Root(_) => None,
|
|
|
|
|
Pkg(pkg_st) => pkg_st.active_pkg_oi().map(Into::into),
|
|
|
|
|
PkgExpr(exprst) => exprst.active_expr_oi().map(Into::into),
|
|
|
|
|
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
|
2023-03-29 09:33:40 -04:00
|
|
|
|
})
|
tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
2023-03-27 11:47:11 -04:00
|
|
|
|
}
|
2023-03-29 13:49:05 -04:00
|
|
|
|
|
|
|
|
|
/// Root an identifier using the [`Self::rooting_oi`] atop of the stack.
|
2023-04-05 10:45:54 -04:00
|
|
|
|
fn defines(&mut self, name: SPair) -> Result<ObjectIndex<Ident>, AsgError> {
|
|
|
|
|
let oi_root = self
|
|
|
|
|
.rooting_oi()
|
|
|
|
|
.ok_or(AsgError::InvalidBindContext(name))?;
|
|
|
|
|
|
|
|
|
|
Ok(self.lookup_lexical_or_missing(name).add_edge_from(
|
|
|
|
|
self.asg_mut(),
|
|
|
|
|
oi_root,
|
|
|
|
|
None,
|
|
|
|
|
))
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Attempt to locate a lexically scoped identifier,
|
|
|
|
|
/// or create a new one if missing.
|
2023-03-29 13:49:05 -04:00
|
|
|
|
///
|
tamer: asg::air: Begin lexical identifier resolution from bottom up
This begins demonstrating that the root will be utilized for identifier
lookup and indexing, as it was originally for TAME and is currently for the
linker.
This was _not_ the original plan---the plan was to have identifiers indexed
only at the package level, at least until we need a global lookup for
something else---but that plan was upended by how externs are currently
handled. So, for now, we need a global scope.
(Externs are resolved by the linker in such a way that _any_ package that
happens to be imported transitively may resolve the import. This is a
global environment, which I had hoped to get rid of, and which will need to
eventually go away (possibly along with externs) to support loading multiple
programs into the graph simultaneously for cross-program analysis.)
This commit renames the base state for `AirAggregate` to emphasize the fact,
especially when observing it in the `AirStack`, and changes
`AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of
the stack upward, rather than reverse, to prove that the system still
operates correctly with this change in place.
The reason for this direction change is to simplify lookup in the most
general case of non-local identifiers, which are almost all of them in
practice---they'll be immediately resolved at the root once they're
indexed. This can be done because I determined that I will _not_ support
shadowing; rationale for that will come later, but TAME is intended to be a
language suitable for non-programmer audiences as well. Note that
identifiers will be resolved lexically within templates in TAMER, unlike
TAME, which means that the expansion context will _not_ be considered when
checking for shadowing, so templates will still be able to compose without a
problem so long as they do not shadow in their definition context. (I'll
have to consider how that affects template-generating templates later on,
but that's an ambiguous construction in TAME today anyway.)
This _does not_ yet index anything at the root where it wasn't already being
indexed explicitly.
DEV-13162
2023-05-10 14:43:33 -04:00
|
|
|
|
/// Since shadowing is not permitted
|
|
|
|
|
/// (but local identifiers are),
|
|
|
|
|
/// we can reduce the cost of lookups for the majority of identifiers
|
|
|
|
|
/// by beginning at the root and continuing down into the narrowest
|
|
|
|
|
/// lexical scope until we find what we're looking for.
|
2023-03-29 13:49:05 -04:00
|
|
|
|
///
|
tamer: asg::air: Begin lexical identifier resolution from bottom up
This begins demonstrating that the root will be utilized for identifier
lookup and indexing, as it was originally for TAME and is currently for the
linker.
This was _not_ the original plan---the plan was to have identifiers indexed
only at the package level, at least until we need a global lookup for
something else---but that plan was upended by how externs are currently
handled. So, for now, we need a global scope.
(Externs are resolved by the linker in such a way that _any_ package that
happens to be imported transitively may resolve the import. This is a
global environment, which I had hoped to get rid of, and which will need to
eventually go away (possibly along with externs) to support loading multiple
programs into the graph simultaneously for cross-program analysis.)
This commit renames the base state for `AirAggregate` to emphasize the fact,
especially when observing it in the `AirStack`, and changes
`AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of
the stack upward, rather than reverse, to prove that the system still
operates correctly with this change in place.
The reason for this direction change is to simplify lookup in the most
general case of non-local identifiers, which are almost all of them in
practice---they'll be immediately resolved at the root once they're
indexed. This can be done because I determined that I will _not_ support
shadowing; rationale for that will come later, but TAME is intended to be a
language suitable for non-programmer audiences as well. Note that
identifiers will be resolved lexically within templates in TAMER, unlike
TAME, which means that the expansion context will _not_ be considered when
checking for shadowing, so templates will still be able to compose without a
problem so long as they do not shadow in their definition context. (I'll
have to consider how that affects template-generating templates later on,
but that's an ambiguous construction in TAME today anyway.)
This _does not_ yet index anything at the root where it wasn't already being
indexed explicitly.
DEV-13162
2023-05-10 14:43:33 -04:00
|
|
|
|
/// Note that the global environment,
|
|
|
|
|
/// represented by the root,
|
|
|
|
|
/// is a pool of identifiers from all packages;
|
|
|
|
|
/// it does not form a hierarchy and local identifiers will not be
|
|
|
|
|
/// indexed outside of their package hierarchy,
|
|
|
|
|
/// so we'll have to continue searching for those.
|
2023-04-05 10:45:54 -04:00
|
|
|
|
fn lookup_lexical_or_missing(&mut self, name: SPair) -> ObjectIndex<Ident> {
|
|
|
|
|
let Self(asg, stack, _) = self;
|
|
|
|
|
|
2023-04-17 15:17:49 -04:00
|
|
|
|
stack
|
2023-04-05 10:45:54 -04:00
|
|
|
|
.iter()
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
.filter_map(|st| st.active_env_oi())
|
2023-04-17 15:17:49 -04:00
|
|
|
|
.find_map(|oi| asg.lookup(oi, name))
|
|
|
|
|
.unwrap_or_else(|| self.create_env_indexed_ident(name))
|
2023-04-05 10:45:54 -04:00
|
|
|
|
}
|
2023-03-29 13:49:05 -04:00
|
|
|
|
|
2023-04-05 10:45:54 -04:00
|
|
|
|
/// Index an identifier within its environment.
|
|
|
|
|
///
|
|
|
|
|
/// TODO: More information as this is formalized.
|
|
|
|
|
fn create_env_indexed_ident(&mut self, name: SPair) -> ObjectIndex<Ident> {
|
2023-05-16 14:52:01 -04:00
|
|
|
|
let Self(asg, stack, _) = self;
|
|
|
|
|
let oi_ident = asg.create(Ident::declare(name));
|
2023-03-31 13:57:11 -04:00
|
|
|
|
|
2023-05-16 14:52:01 -04:00
|
|
|
|
// TODO: This will need the active OI to support `AirIdent`s
|
|
|
|
|
stack
|
|
|
|
|
.iter()
|
|
|
|
|
.rev()
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
.filter_map(|frame| frame.active_env_oi().map(|oi| (oi, frame)))
|
2023-05-16 14:52:01 -04:00
|
|
|
|
.fold(None, |oeoi, (imm_oi, frame)| {
|
|
|
|
|
let eoi_next = oeoi
|
|
|
|
|
.map(|eoi| frame.env_cross_boundary_into(eoi))
|
|
|
|
|
.unwrap_or(EnvScopeKind::Visible(oi_ident));
|
|
|
|
|
|
tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
2023-05-16 23:16:38 -04:00
|
|
|
|
// TODO: Let's find this a better home.
|
|
|
|
|
match eoi_next {
|
|
|
|
|
// There is no use in indexing something that will be
|
|
|
|
|
// filtered out on retrieval.
|
|
|
|
|
EnvScopeKind::Hidden(_) => (),
|
|
|
|
|
_ => asg.index(imm_oi, name, eoi_next),
|
|
|
|
|
}
|
|
|
|
|
|
2023-05-16 14:52:01 -04:00
|
|
|
|
Some(eoi_next)
|
|
|
|
|
});
|
2023-04-05 10:45:54 -04:00
|
|
|
|
|
|
|
|
|
oi_ident
|
2023-03-29 13:49:05 -04:00
|
|
|
|
}
|
tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`. This introduces a layer of abstraction that will allow us to add
the stack.
Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.
This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit. EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before. I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`
everwhere.
DEV-13708
2023-03-27 11:47:11 -04:00
|
|
|
|
}
|
|
|
|
|
|
tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems
from
a) frustration with trying to understand how the system needs to operate
with all of the objects involved; and
b) recognizing that if I'm having difficulty, then others reading the
system later on (including myself) and possibly looking to improve upon
it are going to have a whole lot of trouble.
Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months. This finally begins to formalize
that, out of frustration with package imports. But it will be a weight
lifted off of me as well, with issues of scope always looming.
This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped. Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior. There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.
See the documentation introduced by this commit for more information on
that part of this commit.
Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API. This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with. The
documentation includes clarifying remarks that hopefully make it all
understandable.
DEV-13162
2023-05-12 12:41:51 -04:00
|
|
|
|
/// Property of identifier scope within a given environment.
|
|
|
|
|
///
|
|
|
|
|
/// An _environment_ is the collection of identifiers associated with a
|
|
|
|
|
/// container object.
|
|
|
|
|
/// Environments stack,
|
|
|
|
|
/// such that an environment inherits the identifiers of its parent.
|
|
|
|
|
///
|
|
|
|
|
/// The _scope_ of an identifier is defined by what environments can "see"
|
|
|
|
|
/// that identifier.
|
|
|
|
|
/// For the purposes of TAME's analysis,
|
|
|
|
|
/// we care only about the global environment and shadowing.
|
|
|
|
|
///
|
|
|
|
|
/// The relationship between identifier scope and environment can be
|
|
|
|
|
/// visualized as a two-dimensional table with the environments forming
|
|
|
|
|
/// layers along the x-axes,
|
|
|
|
|
/// and scopes slicing those layers along the y-axies.
|
|
|
|
|
///
|
|
|
|
|
/// TODO: Example visualization.
|
2023-05-16 15:03:54 -04:00
|
|
|
|
///
|
|
|
|
|
/// Root and Global Environment
|
|
|
|
|
/// ===========================
|
|
|
|
|
/// Identifiers are pooled without any defined hierarchy at the root.
|
|
|
|
|
///
|
|
|
|
|
/// An identifier that is part of a pool must be unique.
|
|
|
|
|
/// Since there is no hierarchy,
|
|
|
|
|
/// the system should not suggest that shadowing is not permitted and
|
|
|
|
|
/// should insteam emphasize that such an identifier must be unique
|
|
|
|
|
/// globally.
|
|
|
|
|
///
|
|
|
|
|
/// An identifier's scope can be further refined to provide more useful
|
|
|
|
|
/// diagnostic messages by descending into the package in which it is
|
|
|
|
|
/// defined and evaluating scope relative to the package.
|
2023-05-16 14:52:01 -04:00
|
|
|
|
#[derive(Debug, PartialEq, Copy, Clone)]
|
|
|
|
|
pub(super) enum EnvScopeKind<T = ObjectIndex<Object>> {
|
2023-05-16 15:03:54 -04:00
|
|
|
|
/// This environment owns the identifier,
|
|
|
|
|
/// is descended from an environment that does,
|
|
|
|
|
/// or is a global pool of identifiers.
|
|
|
|
|
Visible(T),
|
tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems
from
a) frustration with trying to understand how the system needs to operate
with all of the objects involved; and
b) recognizing that if I'm having difficulty, then others reading the
system later on (including myself) and possibly looking to improve upon
it are going to have a whole lot of trouble.
Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months. This finally begins to formalize
that, out of frustration with package imports. But it will be a weight
lifted off of me as well, with issues of scope always looming.
This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped. Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior. There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.
See the documentation introduced by this commit for more information on
that part of this commit.
Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API. This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with. The
documentation includes clarifying remarks that hopefully make it all
understandable.
DEV-13162
2023-05-12 12:41:51 -04:00
|
|
|
|
|
|
|
|
|
/// Identifier in this environment is a shadow of a deeper environment.
|
|
|
|
|
///
|
|
|
|
|
/// An identifier is said to cast a shadow on environments higher in its
|
|
|
|
|
/// hierarchy.
|
|
|
|
|
/// Since shadowing is not permitted in TAME,
|
|
|
|
|
/// this can be used to present useful diagnostic information to the
|
|
|
|
|
/// user.
|
|
|
|
|
///
|
|
|
|
|
/// A shadow can be used to check for identifier conflicts,
|
|
|
|
|
/// but it cannot be used for lookup;
|
|
|
|
|
/// this environment should be filtered out of this identifier's
|
|
|
|
|
/// scope.
|
2023-05-16 14:52:01 -04:00
|
|
|
|
Shadow(T),
|
tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems
from
a) frustration with trying to understand how the system needs to operate
with all of the objects involved; and
b) recognizing that if I'm having difficulty, then others reading the
system later on (including myself) and possibly looking to improve upon
it are going to have a whole lot of trouble.
Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months. This finally begins to formalize
that, out of frustration with package imports. But it will be a weight
lifted off of me as well, with issues of scope always looming.
This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped. Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior. There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.
See the documentation introduced by this commit for more information on
that part of this commit.
Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API. This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with. The
documentation includes clarifying remarks that hopefully make it all
understandable.
DEV-13162
2023-05-12 12:41:51 -04:00
|
|
|
|
|
2023-05-16 14:52:01 -04:00
|
|
|
|
/// The identifier is not in scope.
|
|
|
|
|
Hidden(T),
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> EnvScopeKind<T> {
|
|
|
|
|
pub fn into_inner(self) -> T {
|
|
|
|
|
use EnvScopeKind::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
2023-05-16 15:03:54 -04:00
|
|
|
|
Shadow(x) | Visible(x) | Hidden(x) => x,
|
2023-05-16 14:52:01 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Whether this represents an identifier that is in scope.
|
|
|
|
|
pub fn in_scope(self) -> Option<Self> {
|
|
|
|
|
use EnvScopeKind::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
2023-05-16 15:03:54 -04:00
|
|
|
|
Visible(_) => Some(self),
|
2023-05-16 14:52:01 -04:00
|
|
|
|
Shadow(_) | Hidden(_) => None,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> AsRef<T> for EnvScopeKind<T> {
|
|
|
|
|
fn as_ref(&self) -> &T {
|
|
|
|
|
use EnvScopeKind::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
2023-05-16 15:03:54 -04:00
|
|
|
|
Shadow(x) | Visible(x) | Hidden(x) => x,
|
2023-05-16 14:52:01 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T, U> Functor<T, U> for EnvScopeKind<T> {
|
|
|
|
|
type Target = EnvScopeKind<U>;
|
|
|
|
|
|
|
|
|
|
fn map(self, f: impl FnOnce(T) -> U) -> Self::Target {
|
|
|
|
|
use EnvScopeKind::*;
|
|
|
|
|
|
|
|
|
|
match self {
|
|
|
|
|
Shadow(x) => Shadow(f(x)),
|
|
|
|
|
Visible(x) => Visible(f(x)),
|
|
|
|
|
Hidden(x) => Hidden(f(x)),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> From<EnvScopeKind<T>> for Span
|
|
|
|
|
where
|
|
|
|
|
T: Into<Span>,
|
|
|
|
|
{
|
|
|
|
|
fn from(kind: EnvScopeKind<T>) -> Self {
|
|
|
|
|
kind.into_inner().into()
|
|
|
|
|
}
|
tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems
from
a) frustration with trying to understand how the system needs to operate
with all of the objects involved; and
b) recognizing that if I'm having difficulty, then others reading the
system later on (including myself) and possibly looking to improve upon
it are going to have a whole lot of trouble.
Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months. This finally begins to formalize
that, out of frustration with package imports. But it will be a weight
lifted off of me as well, with issues of scope always looming.
This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped. Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior. There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.
See the documentation introduced by this commit for more information on
that part of this commit.
Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API. This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with. The
documentation includes clarifying remarks that hopefully make it all
understandable.
DEV-13162
2023-05-12 12:41:51 -04:00
|
|
|
|
}
|
|
|
|
|
|
2023-03-30 09:20:34 -04:00
|
|
|
|
impl AsMut<AirAggregateCtx> for AirAggregateCtx {
|
|
|
|
|
fn as_mut(&mut self) -> &mut AirAggregateCtx {
|
|
|
|
|
self
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2023-03-29 09:46:17 -04:00
|
|
|
|
impl AsRef<Asg> for AirAggregateCtx {
|
|
|
|
|
fn as_ref(&self) -> &Asg {
|
|
|
|
|
match self {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
Self(asg, _, _) => asg,
|
2023-03-29 09:46:17 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl AsMut<Asg> for AirAggregateCtx {
|
|
|
|
|
fn as_mut(&mut self) -> &mut Asg {
|
|
|
|
|
match self {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
Self(asg, _, _) => asg,
|
2023-03-29 09:46:17 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl AsMut<AirStack> for AirAggregateCtx {
|
|
|
|
|
fn as_mut(&mut self) -> &mut AirStack {
|
|
|
|
|
match self {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
Self(_, stack, _) => stack,
|
2023-03-29 09:46:17 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<AirAggregateCtx> for Asg {
|
|
|
|
|
fn from(ctx: AirAggregateCtx) -> Self {
|
|
|
|
|
match ctx {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
AirAggregateCtx(asg, _, _) => asg,
|
2023-03-29 09:46:17 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl From<Asg> for AirAggregateCtx {
|
|
|
|
|
fn from(asg: Asg) -> Self {
|
2023-03-30 22:28:22 -04:00
|
|
|
|
Self(asg, Default::default(), None)
|
2023-03-29 09:46:17 -04:00
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`. This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate. I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore. I don't like making large changes such
as this one.
There is still work to do here. First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.
Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`. The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.
Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well. Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.
That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together. Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.
DEV-11864
2022-05-27 13:51:29 -04:00
|
|
|
|
#[cfg(test)]
|
2023-05-17 10:55:06 -04:00
|
|
|
|
pub mod test;
|