tame/tamer/src/asg/air.rs

922 lines
31 KiB
Rust
Raw Normal View History

tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
// ASG IR
//
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
//
// This file is part of TAME.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
//! Intermediate representation for construction of the
//! [abstract semantic graph (ASG)](super) (AIR).
//!
//! AIR serves as an abstraction layer between higher-level parsers and the
//! aggregate ASG.
//! It allows parsers to operate as a raw stream of data without having to
//! worry about ownership of or references to the ASG,
//! and allows for multiple such parsers to be joined.
//!
//! AIR is _not_ intended to replace the API of the ASG---it
//! is intended as a termination point for the parsing pipeline,
//! and as such implements a subset of the ASG's API that is suitable
//! for aggregating raw data from source and object files.
//! Given that it does so little and is so close to the [`Asg`] API,
//! one might say that the abstraction is as light as air,
//! but that would surely result in face-palming and so we're not going
//! air such cringeworthy dad jokes here.
use super::{
graph::object::{Object, ObjectIndexTo, ObjectIndexToTree, Pkg, Root, Tpl},
Asg, AsgError, Expr, Ident, ObjectIndex,
};
use crate::{
diagnose::Annotate,
f::Functor,
parse::{prelude::*, StateStack},
span::Span,
sym::SymbolId,
};
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
use std::fmt::{Debug, Display};
#[cfg(test)]
use super::{graph::object::ObjectRelatable, ObjectIndexRelTo};
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
#[macro_use]
mod ir;
pub use ir::Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
mod expr;
mod opaque;
mod pkg;
mod tpl;
use expr::AirExprAggregate;
use opaque::AirOpaqueAggregate;
use pkg::AirPkgAggregate;
use tpl::AirTplAggregate;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
pub type IdentSym = SymbolId;
pub type DepSym = SymbolId;
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// AIR parser state.
#[derive(Debug, PartialEq, Default)]
pub enum AirAggregate {
/// Parser has not yet been initialized.
#[default]
Uninit,
tamer: asg::air: Begin lexical identifier resolution from bottom up This begins demonstrating that the root will be utilized for identifier lookup and indexing, as it was originally for TAME and is currently for the linker. This was _not_ the original plan---the plan was to have identifiers indexed only at the package level, at least until we need a global lookup for something else---but that plan was upended by how externs are currently handled. So, for now, we need a global scope. (Externs are resolved by the linker in such a way that _any_ package that happens to be imported transitively may resolve the import. This is a global environment, which I had hoped to get rid of, and which will need to eventually go away (possibly along with externs) to support loading multiple programs into the graph simultaneously for cross-program analysis.) This commit renames the base state for `AirAggregate` to emphasize the fact, especially when observing it in the `AirStack`, and changes `AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of the stack upward, rather than reverse, to prove that the system still operates correctly with this change in place. The reason for this direction change is to simplify lookup in the most general case of non-local identifiers, which are almost all of them in practice---they'll be immediately resolved at the root once they're indexed. This can be done because I determined that I will _not_ support shadowing; rationale for that will come later, but TAME is intended to be a language suitable for non-programmer audiences as well. Note that identifiers will be resolved lexically within templates in TAMER, unlike TAME, which means that the expansion context will _not_ be considered when checking for shadowing, so templates will still be able to compose without a problem so long as they do not shadow in their definition context. (I'll have to consider how that affects template-generating templates later on, but that's an ambiguous construction in TAME today anyway.) This _does not_ yet index anything at the root where it wasn't already being indexed explicitly. DEV-13162
2023-05-10 14:43:33 -04:00
/// Parser is in the root context.
///
/// As a parser,
/// this does nothing but await work.
/// Its presence in the [`AirStack`] is used for the global environment.
Root(ObjectIndex<Root>),
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
/// Parsing a package.
Pkg(AirPkgAggregate),
/// Parsing an expression.
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
///
/// This expects to inherit an [`AirExprAggregate`] from the prior state
/// so that we are not continuously re-allocating its stack for each
/// new expression root.
PkgExpr(AirExprAggregate),
/// Parser is in template parsing mode.
///
/// All objects encountered until the closing [`Air::TplEnd`] will be
/// parented to this template rather than the parent [`Pkg`].
/// See [`Air::TplStart`] for more information.
PkgTpl(AirTplAggregate),
/// Parsing opaque objects.
///
/// This parser is intended for loading declarations from object files
/// without loading their corresponding definitions.
PkgOpaque(AirOpaqueAggregate),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
impl Display for AirAggregate {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
use AirAggregate::*;
match self {
Uninit => write!(f, "awaiting AIR input"),
Root(_) => write!(f, "awaiting input at root"),
Pkg(pkg) => {
write!(f, "defining a package: {pkg}")
}
PkgExpr(expr) => {
write!(f, "defining a package expression: {expr}")
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
PkgTpl(tpl) => {
write!(f, "building a template: {tpl}")
}
PkgOpaque(opaque) => {
write!(f, "loading opaque objects: {opaque}")
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
impl From<AirPkgAggregate> for AirAggregate {
fn from(st: AirPkgAggregate) -> Self {
Self::Pkg(st)
}
}
impl From<AirExprAggregate> for AirAggregate {
fn from(st: AirExprAggregate) -> Self {
Self::PkgExpr(st)
}
}
impl From<AirTplAggregate> for AirAggregate {
fn from(st: AirTplAggregate) -> Self {
Self::PkgTpl(st)
}
}
impl From<AirOpaqueAggregate> for AirAggregate {
fn from(st: AirOpaqueAggregate) -> Self {
Self::PkgOpaque(st)
}
}
impl ParseState for AirAggregate {
type Token = Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
type Object = ();
type Error = AsgError;
type Context = AirAggregateCtx;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
fn parse_token(
self,
tok: Self::Token,
ctx: &mut Self::Context,
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
) -> crate::parse::TransitionResult<Self> {
use ir::{AirSubsets::*, AirTodo::*};
use AirAggregate::*;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
match (self, tok.into()) {
// Initialize the parser with the graph root,
// or continue with a previous context that has already been
// initialized.
// See `asg::air::test::resume_previous_parsing_context` for an
// explanation of why this is important.
(Uninit, tok) => {
let oi_root = ctx.asg_ref().root(tok.span());
ctx.stack().continue_or_init(|| Root(oi_root), tok)
}
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
(st, AirTodo(Todo(_))) => Transition(st).incomplete(),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
// Package
//
// Note that `ret_or_transfer` will return from the active frame
// if it is in an accepting state,
// and so encountering a properly nested `PkgClose` will pop
// frames off of the stack until reaching the still-active
// parent package frame.
(
st @ (Root(..) | PkgExpr(..) | PkgTpl(..) | PkgOpaque(..)),
tok @ AirPkg(..),
) => ctx.ret_or_transfer(st, tok, AirPkgAggregate::new()),
(Pkg(pkg), AirPkg(etok)) => ctx.proxy(pkg, etok),
(Pkg(pkg), AirBind(etok)) => ctx.proxy(pkg, etok),
(Pkg(pkg), AirDoc(etok)) => ctx.proxy(pkg, etok),
// Expression
(st @ (Pkg(_) | PkgTpl(_)), tok @ AirExpr(..)) => {
ctx.ret_or_transfer(st, tok, AirExprAggregate::new())
}
(PkgExpr(expr), AirExpr(etok)) => ctx.proxy(expr, etok),
(PkgExpr(expr), AirBind(etok)) => ctx.proxy(expr, etok),
(PkgExpr(expr), AirDoc(etok)) => ctx.proxy(expr, etok),
// Template
(st @ (Pkg(_) | PkgExpr(_)), tok @ AirTpl(..)) => {
ctx.ret_or_transfer(st, tok, AirTplAggregate::new())
}
(PkgTpl(tplst), AirTpl(ttok)) => ctx.proxy(tplst, ttok),
(PkgTpl(tplst), AirBind(ttok)) => ctx.proxy(tplst, ttok),
(PkgTpl(tplst), AirDoc(ttok)) => ctx.proxy(tplst, ttok),
// Opaque
//
// By having opaque object loading be its _own_ child parser,
// we ensure that the active package frame becomes held on the
// stack before loading e.g. opaque identifiers.
// Since scope is determined by stack frames,
// this has the effect of ensuring that the package `st`
// becomes included in the identifier's scope.
(st @ Pkg(_), tok @ AirIdent(..)) => {
ctx.ret_or_transfer(st, tok, AirOpaqueAggregate::new())
}
(PkgOpaque(opaque), AirIdent(otok)) => ctx.proxy(opaque, otok),
(
PkgOpaque(_),
tok @ (AirExpr(..) | AirBind(..) | AirTpl(..) | AirDoc(..)),
) => {
// This is simply not expected at the time of writing,
// since this is used for importing object files.
crate::diagnostic_panic!(
vec![
tok.span()
.internal_error("this is not an opaque identifier"),
tok.span().help(
"this may represent a problem with an object file"
)
],
"expected opaque identifier, found {tok}",
);
}
(
st @ Root(_),
tok @ (AirExpr(..) | AirBind(..) | AirTpl(..) | AirDoc(..)),
) => Transition(st).err(AsgError::PkgExpected(tok.span())),
(st @ (Root(..) | PkgExpr(..) | PkgTpl(..)), AirIdent(tok)) => {
Transition(st).err(AsgError::UnexpectedOpaqueIdent(tok.name()))
}
}
}
fn is_accepting(&self, ctx: &Self::Context) -> bool {
ctx.stack_ref().iter().all(|st| st.active_is_accepting(ctx))
&& self.active_is_accepting(ctx)
}
}
impl AirAggregate {
/// Whether the active parser is completed with active parsing.
///
/// This method is used to determine whether control ought to be
/// transferred to a new child parser.
///
/// If a child parser is active,
/// then its [`ParseState::is_accepting`] will be consulted.
fn active_is_complete(&self, ctx: &<Self as ParseState>::Context) -> bool {
use AirAggregate::*;
match self {
Uninit => false,
// We can't be done with something we're not doing.
// This is necessary to start the first child parser.
Root(_) => false,
Pkg(st) => st.is_accepting(ctx),
PkgExpr(st) => st.is_accepting(ctx),
PkgTpl(st) => st.is_accepting(ctx),
PkgOpaque(st) => st.is_accepting(ctx),
}
}
// Whether the parser is in an accepting state.
fn active_is_accepting(&self, ctx: &<Self as ParseState>::Context) -> bool {
use AirAggregate::*;
match self {
Uninit => false,
// This must not recurse on `AirAggregate::is_accepting`,
// otherwise it'll be mutually recursive.
Root(_) => true,
Pkg(st) => st.is_accepting(ctx),
PkgExpr(st) => st.is_accepting(ctx),
PkgTpl(st) => st.is_accepting(ctx),
PkgOpaque(st) => st.is_accepting(ctx),
}
}
/// The rooting context for [`Ident`]s for the active parser.
///
/// A value of [`None`] indicates that the current parser does not
/// support direct bindings,
/// but a parent context may
/// (see [`AirAggregateCtx::rooting_oi`]).
fn active_rooting_oi(&self) -> Option<ObjectIndexToTree<Ident>> {
use AirAggregate::*;
match self {
Uninit => None,
// Root will serve as a pool of identifiers,
// but it can never _contain_ their definitions.
// See `active_env_oi`.
Root(_) => None,
// Packages always serve as roots for identifiers
// (that is their entire purpose).
Pkg(pkgst) => pkgst.active_pkg_oi().map(Into::into),
// Expressions never serve as roots for identifiers;
// this will always fall through to the parent context.
// Since the parent context is a package or a template,
// the next frame should succeed.
PkgExpr(_) => None,
// Identifiers bound while within a template definition context
// must bind to the eventual _expansion_ site,
// as if the body were pasted there.
// Templates must therefore serve as containers for identifiers
// bound therein.
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
// Loading of opaque objects happens within the context of the
// parent frame.
// At the time of writing,
// that is only a package.
PkgOpaque(_) => None,
}
}
/// Active environment for identifier lookups.
///
/// An environment is a superset of a container,
/// which is described by [`Self::active_rooting_oi`].
/// For example,
/// [`Self::Root`] cannot own any identifiers,
/// but it can serve as a pool of references to them.
fn active_env_oi(&self) -> Option<ObjectIndexTo<Ident>> {
use AirAggregate::*;
match self {
Root(oi_root) => Some((*oi_root).into()),
_ => self.active_rooting_oi().map(Into::into),
}
}
/// Adjust a [`EnvScopeKind`] while crossing an environment boundary
/// into `self`.
///
/// An identifier is _visible_ at the environment in which it is defined.
/// This identifier casts a _shadow_ to lower environments,
/// with the exception of the root.
/// The _root_ will absorb adjacent visible identifiers into a _pool_,
/// which is distinct from the hierarchy that is otherwise created at
/// the package level and lower.
fn env_cross_boundary_into<T>(
&self,
kind: EnvScopeKind<T>,
) -> EnvScopeKind<T> {
use AirAggregate::*;
use EnvScopeKind::*;
match (self, kind) {
// This is not an environment.
(Uninit, kind) => kind,
// This is just a parsing state,
// not an environment.
(PkgOpaque(_), kind) => kind,
// Hidden is a fixpoint.
(_, kind @ Hidden(_)) => kind,
// Expressions do not introduce their own environment
// (they are not containers)
// and so act as an identity function.
(PkgExpr(_), kind) => kind,
// A visible identifier will always cast a shadow in one step.
// A shadow will always be cast (propagate) until the root.
(Pkg(_) | PkgTpl(_), Visible(x) | Shadow(x)) => Shadow(x),
// Above we see that Visual will always transition to Shadow in
// one step.
// Consequently,
// Visible at Root means that we're a package-level Visible,
// which must contribute to the pool.
(Root(_), Visible(x)) => Visible(x),
// If we're _not_ Visible at the root,
// then we're _not_ a package-level definition,
// and so we should _not_ contribute to the pool.
(Root(_), Shadow(x)) => Hidden(x),
}
}
}
/// Additional parser context,
/// including the ASG and parser stack frames.
#[derive(Debug, Default)]
pub struct AirAggregateCtx {
/// The ASG under construction by this parser.
///
/// The ASG must be under exclusive ownership of this parser to ensure
/// that graph metadata
/// (e.g. indexes)
/// are accurate.
asg: Asg,
/// Held parser frames.
stack: AirStack,
/// The package currently being parsed,
/// if any.
///
/// This is not necessarily the current compilation unit,
/// as the parser may be parsing imports.
ooi_pkg: Option<ObjectIndex<Pkg>>,
}
/// Limit of the maximum number of held parser frames.
///
/// Note that this is the number of [`ParseState`]s held,
/// _not_ the depth of the graph at a given point.
/// The intent of this is to limit runaway recursion in the event of some
/// bug in the system;
/// while the input stream is certainly finite,
/// lookahead tokens cause recursion that does not provably
/// terminate.
///
/// This limit is arbitrarily large,
/// but hopefully such that no legitimate case will ever hit it.
const MAX_AIR_STACK_DEPTH: usize = 1024;
/// Held parser stack frames.
///
/// See [`AirAggregateCtx`] for more information.
pub type AirStack = StateStack<AirAggregate, MAX_AIR_STACK_DEPTH>;
impl AirAggregateCtx {
fn asg_mut(&mut self) -> &mut Asg {
self.as_mut()
}
fn asg_ref(&self) -> &Asg {
self.as_ref()
}
fn stack(&mut self) -> &mut AirStack {
&mut self.stack
}
fn stack_ref(&self) -> &AirStack {
&self.stack
}
/// Return control to the parser atop of the stack if `st` is an
/// accepting state,
/// otherwise transfer control to a new parser `to`.
///
/// This serves as a balance with the behavior of [`Self::proxy`].
/// Rather than checking for an accepting state after each proxy,
/// or having the child parsers return to the top stack frame once
/// they have completed,
/// we leave the child parser in place to potentially handle more
/// tokens of the same type.
/// For example,
/// adjacent expressions can re-use the same parser rather than having
/// to pop and push for each sibling.
///
/// Consequently,
/// this means that a parser may be complete when we need to push and
/// transfer control to another parser.
/// Before pushing,
/// we first check to see if the parser atop of the stack is in an
/// accepting state.
/// If so,
/// then we are a sibling,
/// and so instead of proceeding with instantiating a new parser,
/// we return to the one atop of the stack and delegate to it.
///
/// If `st` is _not_ in an accepting state,
/// that means that we are a _child_;
/// we then set aside the state `st` on the stack and transfer
/// control to the child `to`.
///
/// See also [`Self::proxy`].
fn ret_or_transfer<S: Into<AirAggregate>, SB: Into<AirAggregate>>(
&mut self,
st: S,
tok: impl Token + Into<Air>,
to: SB,
) -> TransitionResult<AirAggregate> {
let st_super = st.into();
if st_super.active_is_complete(self) {
// TODO: error (this should never happen, so maybe panic instead?)
self.stack().ret_or_dead(AirAggregate::Uninit, tok)
} else {
self.stack().transfer_with_ret(
Transition(st_super),
Transition(to.into()).incomplete().with_lookahead(tok),
)
}
}
/// Proxy `tok` to `st`,
/// returning to the state atop of the stack if parsing reaches a dead
/// state.
///
/// See also [`Self::ret_or_transfer`].
fn proxy<S: ParseState<Super = AirAggregate, Context = Self>>(
&mut self,
st: S,
tok: impl Token + Into<S::Token>,
) -> TransitionResult<AirAggregate> {
st.delegate_child(tok.into(), self, |_deadst, tok, ctx| {
// TODO: error (this should never happen, so maybe panic instead?)
ctx.stack().ret_or_dead(AirAggregate::Uninit, tok)
})
}
/// Create a new rooted package of the given canonical name and record
/// it as the active package.
///
/// A canonical package name is a path relative to the project root.
///
/// This operation will fail if a package of the same name has already
/// been declared.
fn pkg_begin(
&mut self,
start: Span,
name: SPair,
) -> Result<ObjectIndex<Pkg>, AsgError> {
let Self {
asg, ooi_pkg: pkg, ..
} = self;
let oi_root = asg.root(start);
let oi_pkg = asg.create(Pkg::new_canonical(start, name)?);
let eoi_pkg = EnvScopeKind::Visible(oi_pkg);
asg.try_index(oi_root, name, eoi_pkg).map_err(|oi_prev| {
let prev = oi_prev.resolve(asg);
// unwrap note: a canonical name must exist for this error to
// have been thrown,
// but this will at least not blow up if something really
// odd happens.
AsgError::PkgRedeclare(prev.canonical_name(), name)
})?;
oi_pkg.root(asg);
pkg.replace(oi_pkg);
Ok(oi_pkg)
}
/// Indicate that there is no longer any active package.
fn pkg_clear(&mut self) {
self.ooi_pkg.take();
}
/// The active package if any.
fn pkg_oi(&self) -> Option<ObjectIndex<Pkg>> {
self.ooi_pkg
}
/// The active container (rooting context) for [`Ident`]s.
///
/// The integer value returned represents the stack offset at which the
/// rooting index was found,
/// with `0` representing the package.
///
/// A value of [`None`] indicates that no bindings are permitted in the
/// current context.
fn rooting_oi(&self) -> Option<ObjectIndexToTree<Ident>> {
self.stack
.iter()
.rev()
.find_map(|st| st.active_rooting_oi())
}
/// The active dangling expression context for [`Expr`]s.
///
/// A value of [`None`] indicates that expressions are not permitted to
/// dangle in the current context
/// (and so must be identified).
fn dangling_expr_oi(&self) -> Option<ObjectIndexTo<Expr>> {
use AirAggregate::*;
self.stack.iter().rev().find_map(|st| match st {
Uninit => None,
// It should never be possible to define expressions directly in
// Root.
Root(_) => None,
// A dangling expression in a package context would be
// unreachable.
// There should be no parent frame and so this will fail to find
// a value.
Pkg(_) => None,
// Expressions may always contain other expressions,
// and so this method should not be consulted in such a
// context.
// Nonetheless,
// fall through to the parent frame and give a correct answer.
PkgExpr(_) => None,
// Templates serve as containers for dangling expressions,
// since they may expand into an context where they are not
// considered to be dangling.
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
// Expressions are transparent definitions,
// not opaque,
// and so not permitted in this context.
PkgOpaque(_) => None,
})
}
/// The active expansion target (splicing context) for [`Tpl`]s.
///
/// A value of [`None`] indicates that template expansion is not
/// permitted in this current context.
fn expansion_oi(&self) -> Option<ObjectIndexTo<Tpl>> {
use AirAggregate::*;
self.stack.iter().rev().find_map(|st| match st {
Uninit => None,
Root(_) => None,
Pkg(pkg_st) => pkg_st.active_pkg_oi().map(Into::into),
PkgExpr(exprst) => exprst.active_expr_oi().map(Into::into),
PkgTpl(tplst) => tplst.active_tpl_oi().map(Into::into),
// Templates _could_ conceptually expand into opaque objects,
// but the source language of TAME provides no mechanism to do
// such a thing,
// and so it'd be best to leave this alone unless it's
// actually needed.
PkgOpaque(_) => None,
})
}
/// Root an identifier using the [`Self::rooting_oi`] atop of the stack.
fn defines(&mut self, name: SPair) -> Result<ObjectIndex<Ident>, AsgError> {
let oi_root = self
.rooting_oi()
.ok_or(AsgError::InvalidBindContext(name))?;
Ok(self.lookup_lexical_or_missing(name).add_edge_from(
self.asg_mut(),
oi_root,
None,
))
}
/// Attempt to retrieve an identifier and its scope information from the
/// graph by name relative to the environment `env`.
///
/// See [`Self::lookup`] for more information.
#[cfg(test)]
fn env_scope_lookup_raw<O: ObjectRelatable>(
&self,
env: impl ObjectIndexRelTo<O>,
name: SPair,
) -> Option<EnvScopeKind<ObjectIndex<O>>> {
self.asg_ref().lookup_raw(env, name)
}
/// Resolve an identifier at the scope of the provided environment.
///
/// If the identifier is not in scope at `env`,
/// [`None`] will be returned.
#[cfg(test)]
fn env_scope_lookup<O: ObjectRelatable>(
&self,
env: impl ObjectIndexRelTo<O>,
name: SPair,
) -> Option<ObjectIndex<O>> {
self.env_scope_lookup_raw(env, name)
.and_then(EnvScopeKind::in_scope)
.map(EnvScopeKind::into_inner)
}
/// Attempt to locate a lexically scoped identifier in the current stack,
/// or create a new one if missing.
///
tamer: asg::air: Begin lexical identifier resolution from bottom up This begins demonstrating that the root will be utilized for identifier lookup and indexing, as it was originally for TAME and is currently for the linker. This was _not_ the original plan---the plan was to have identifiers indexed only at the package level, at least until we need a global lookup for something else---but that plan was upended by how externs are currently handled. So, for now, we need a global scope. (Externs are resolved by the linker in such a way that _any_ package that happens to be imported transitively may resolve the import. This is a global environment, which I had hoped to get rid of, and which will need to eventually go away (possibly along with externs) to support loading multiple programs into the graph simultaneously for cross-program analysis.) This commit renames the base state for `AirAggregate` to emphasize the fact, especially when observing it in the `AirStack`, and changes `AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of the stack upward, rather than reverse, to prove that the system still operates correctly with this change in place. The reason for this direction change is to simplify lookup in the most general case of non-local identifiers, which are almost all of them in practice---they'll be immediately resolved at the root once they're indexed. This can be done because I determined that I will _not_ support shadowing; rationale for that will come later, but TAME is intended to be a language suitable for non-programmer audiences as well. Note that identifiers will be resolved lexically within templates in TAMER, unlike TAME, which means that the expansion context will _not_ be considered when checking for shadowing, so templates will still be able to compose without a problem so long as they do not shadow in their definition context. (I'll have to consider how that affects template-generating templates later on, but that's an ambiguous construction in TAME today anyway.) This _does not_ yet index anything at the root where it wasn't already being indexed explicitly. DEV-13162
2023-05-10 14:43:33 -04:00
/// Since shadowing is not permitted
/// (but local identifiers are),
/// we can reduce the cost of lookups for the majority of identifiers
/// by beginning at the root and continuing down into the narrowest
/// lexical scope until we find what we're looking for.
///
tamer: asg::air: Begin lexical identifier resolution from bottom up This begins demonstrating that the root will be utilized for identifier lookup and indexing, as it was originally for TAME and is currently for the linker. This was _not_ the original plan---the plan was to have identifiers indexed only at the package level, at least until we need a global lookup for something else---but that plan was upended by how externs are currently handled. So, for now, we need a global scope. (Externs are resolved by the linker in such a way that _any_ package that happens to be imported transitively may resolve the import. This is a global environment, which I had hoped to get rid of, and which will need to eventually go away (possibly along with externs) to support loading multiple programs into the graph simultaneously for cross-program analysis.) This commit renames the base state for `AirAggregate` to emphasize the fact, especially when observing it in the `AirStack`, and changes `AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of the stack upward, rather than reverse, to prove that the system still operates correctly with this change in place. The reason for this direction change is to simplify lookup in the most general case of non-local identifiers, which are almost all of them in practice---they'll be immediately resolved at the root once they're indexed. This can be done because I determined that I will _not_ support shadowing; rationale for that will come later, but TAME is intended to be a language suitable for non-programmer audiences as well. Note that identifiers will be resolved lexically within templates in TAMER, unlike TAME, which means that the expansion context will _not_ be considered when checking for shadowing, so templates will still be able to compose without a problem so long as they do not shadow in their definition context. (I'll have to consider how that affects template-generating templates later on, but that's an ambiguous construction in TAME today anyway.) This _does not_ yet index anything at the root where it wasn't already being indexed explicitly. DEV-13162
2023-05-10 14:43:33 -04:00
/// Note that the global environment,
/// represented by the root,
/// is a pool of identifiers from all packages;
/// it does not form a hierarchy and local identifiers will not be
/// indexed outside of their package hierarchy,
/// so we'll have to continue searching for those.
tamer: asg::air: Root AirIdent operations using AirAggregateCtx This is the culmination of a great deal of work over the past few weeks. Indeed, this change has been prototyped a number of different ways and has lived in a stash of mine, in one form or another, for a few weeks. This is not done just yet---I have to finish moving the index out of Asg, and then clean up a little bit more---but this is a significant simplification of the system. It was very difficult to reason about prior approaches, and this finally moves toward doing something that I wasn't sure if I'd be able to do successfully: formalize scope using AirAggregate's stack and encapsulate indexing as something that is _supplemental_ to the graph, rather than an integral component of it. This _does not yet_ index the AirIdent operation on the package itself because the active state is not part of the stack; that is one of the remaining changes I still have stashed. It will be needed shortly for package imports. This rationale will have to appear in docs, which I intend to write soon, but: this means that `Asg` contains _resolved_ data and itself has no concept of scope. The state of the ASG immediately after parsing _can_ be used to derive what the scope _must_ be (and indeed that's what `asg::air::test::scope::derive_scopes_from_asg` does), but once we start performing optimizations, that will no longer be true in all cases. This means that lexical scope is a property of parsing, which, well, seems kind of obvious from its name. But the awkwardness was that, if we consider scope to be purely a parse-time thing---used only to construct the relationships on the graph and then be discarded---then how do we query for information on the graph? We'd have to walk the graph in search of an identifier, which is slow. But when do we need to do such a thing? For tests, it doesn't matter if it's a little bit slow, and the graphs aren't all that large. And for operations like template expansion and optimizations, if they need access to a particular index, then we'll be sure to generate or provide the appropriate one. If we need a central database of identifiers for tooling in the future, we'll create one then. No general-purpose identifier lookup _is_ actually needed. And with that, `Asg::lookup_or_missing` is removed. It has been around since the beginning of the ASG, when the linker was just a prototype, so it's the end of TAMER's early era as I was trying to discover exactly what I wanted the ASG to represent. DEV-13162
2023-05-17 11:57:46 -04:00
///
/// The provided name's span is used to seed the missing object with
/// some sort of context to aid in debugging why a missing object
/// was introduced to the graph.
/// The provided span will be used by the returned [`ObjectIndex`] even
/// if an object exists on the graph,
/// which can be used for retaining information on the location that
/// requested the object.
/// To retrieve the span of a previously declared object,
/// you must resolve the [`ObjectIndex`] and inspect it.
fn lookup_lexical_or_missing(&mut self, name: SPair) -> ObjectIndex<Ident> {
let Self { asg, stack, .. } = self;
stack
.iter()
.filter_map(|st| st.active_env_oi())
.find_map(|oi| asg.lookup(oi, name))
.unwrap_or_else(|| self.create_env_indexed_ident(name))
}
/// Index an identifier within its environment.
///
/// TODO: More information as this is formalized.
fn create_env_indexed_ident(&mut self, name: SPair) -> ObjectIndex<Ident> {
let Self { asg, stack, .. } = self;
let oi_ident = asg.create(Ident::declare(name));
stack
.iter()
.rev()
.filter_map(|frame| frame.active_env_oi().map(|oi| (oi, frame)))
.fold(None, |oeoi, (imm_oi, frame)| {
let eoi_next = oeoi
.map(|eoi| frame.env_cross_boundary_into(eoi))
.unwrap_or(EnvScopeKind::Visible(oi_ident));
// TODO: Let's find this a better home.
match eoi_next {
// There is no use in indexing something that will be
// filtered out on retrieval.
EnvScopeKind::Hidden(_) => (),
_ => asg.index(imm_oi, name, eoi_next),
}
Some(eoi_next)
});
oi_ident
}
/// Consume the context and yield the inner [`Asg`].
///
/// This indicates that all package parsing has been completed and that
/// the ASG contains complete information about the program sources
/// for the requested compilation unit.
pub fn finish(self) -> Asg {
self.asg
}
}
tamer: asg::air: Begin to introduce explicit scope testing There's a lot of documentation on this in the commit itself, but this stems from a) frustration with trying to understand how the system needs to operate with all of the objects involved; and b) recognizing that if I'm having difficulty, then others reading the system later on (including myself) and possibly looking to improve upon it are going to have a whole lot of trouble. Identifier scope is something I've been mulling over for years, and more formally for the past couple of months. This finally begins to formalize that, out of frustration with package imports. But it will be a weight lifted off of me as well, with issues of scope always looming. This demonstrates a declarative means of testing for scope by scanning the entire graph in tests to determine where an identifier has been scoped. Since no such scoping has been implemented yet, the tests demonstrate how they will look, but otherwise just test for current behavior. There is more existing behavior to check, and further there will be _references_ to check, as they'll also leave a trail of scope indexing behind as part of the resolution process. See the documentation introduced by this commit for more information on that part of this commit. Introducing the graph scanning, with the ASG's static assurances, required more lowering of dynamic types into the static types required by the API. This was itself a confusing challenge that, while not all that bad in retrospect, was something that I initially had some trouble with. The documentation includes clarifying remarks that hopefully make it all understandable. DEV-13162
2023-05-12 12:41:51 -04:00
/// Property of identifier scope within a given environment.
///
/// An _environment_ is the collection of identifiers associated with a
/// container object.
/// Environments stack,
/// such that an environment inherits the identifiers of its parent.
///
/// The _scope_ of an identifier is defined by what environments can "see"
/// that identifier.
/// For the purposes of TAME's analysis,
/// we care only about the global environment and shadowing.
///
/// The relationship between identifier scope and environment can be
/// visualized as a two-dimensional table with the environments forming
/// layers along the x-axes,
/// and scopes slicing those layers along the y-axies.
///
/// TODO: Example visualization.
///
/// Root and Global Environment
/// ===========================
/// Identifiers are pooled without any defined hierarchy at the root.
///
/// An identifier that is part of a pool must be unique.
/// Since there is no hierarchy,
/// the system should not suggest that shadowing is not permitted and
/// should insteam emphasize that such an identifier must be unique
/// globally.
///
/// An identifier's scope can be further refined to provide more useful
/// diagnostic messages by descending into the package in which it is
/// defined and evaluating scope relative to the package.
#[derive(Debug, PartialEq, Copy, Clone)]
pub(super) enum EnvScopeKind<T = ObjectIndex<Object>> {
/// This environment owns the identifier,
/// is descended from an environment that does,
/// or is a global pool of identifiers.
Visible(T),
tamer: asg::air: Begin to introduce explicit scope testing There's a lot of documentation on this in the commit itself, but this stems from a) frustration with trying to understand how the system needs to operate with all of the objects involved; and b) recognizing that if I'm having difficulty, then others reading the system later on (including myself) and possibly looking to improve upon it are going to have a whole lot of trouble. Identifier scope is something I've been mulling over for years, and more formally for the past couple of months. This finally begins to formalize that, out of frustration with package imports. But it will be a weight lifted off of me as well, with issues of scope always looming. This demonstrates a declarative means of testing for scope by scanning the entire graph in tests to determine where an identifier has been scoped. Since no such scoping has been implemented yet, the tests demonstrate how they will look, but otherwise just test for current behavior. There is more existing behavior to check, and further there will be _references_ to check, as they'll also leave a trail of scope indexing behind as part of the resolution process. See the documentation introduced by this commit for more information on that part of this commit. Introducing the graph scanning, with the ASG's static assurances, required more lowering of dynamic types into the static types required by the API. This was itself a confusing challenge that, while not all that bad in retrospect, was something that I initially had some trouble with. The documentation includes clarifying remarks that hopefully make it all understandable. DEV-13162
2023-05-12 12:41:51 -04:00
/// Identifier in this environment is a shadow of a deeper environment.
///
/// An identifier is said to cast a shadow on environments higher in its
/// hierarchy.
/// Since shadowing is not permitted in TAME,
/// this can be used to present useful diagnostic information to the
/// user.
///
/// A shadow can be used to check for identifier conflicts,
/// but it cannot be used for lookup;
/// this environment should be filtered out of this identifier's
/// scope.
Shadow(T),
tamer: asg::air: Begin to introduce explicit scope testing There's a lot of documentation on this in the commit itself, but this stems from a) frustration with trying to understand how the system needs to operate with all of the objects involved; and b) recognizing that if I'm having difficulty, then others reading the system later on (including myself) and possibly looking to improve upon it are going to have a whole lot of trouble. Identifier scope is something I've been mulling over for years, and more formally for the past couple of months. This finally begins to formalize that, out of frustration with package imports. But it will be a weight lifted off of me as well, with issues of scope always looming. This demonstrates a declarative means of testing for scope by scanning the entire graph in tests to determine where an identifier has been scoped. Since no such scoping has been implemented yet, the tests demonstrate how they will look, but otherwise just test for current behavior. There is more existing behavior to check, and further there will be _references_ to check, as they'll also leave a trail of scope indexing behind as part of the resolution process. See the documentation introduced by this commit for more information on that part of this commit. Introducing the graph scanning, with the ASG's static assurances, required more lowering of dynamic types into the static types required by the API. This was itself a confusing challenge that, while not all that bad in retrospect, was something that I initially had some trouble with. The documentation includes clarifying remarks that hopefully make it all understandable. DEV-13162
2023-05-12 12:41:51 -04:00
/// The identifier is not in scope.
Hidden(T),
}
impl<T> EnvScopeKind<T> {
pub fn into_inner(self) -> T {
use EnvScopeKind::*;
match self {
Shadow(x) | Visible(x) | Hidden(x) => x,
}
}
/// Whether this represents an identifier that is in scope.
pub fn in_scope(self) -> Option<Self> {
use EnvScopeKind::*;
match self {
Visible(_) => Some(self),
Shadow(_) | Hidden(_) => None,
}
}
}
impl<T> AsRef<T> for EnvScopeKind<T> {
fn as_ref(&self) -> &T {
use EnvScopeKind::*;
match self {
Shadow(x) | Visible(x) | Hidden(x) => x,
}
}
}
impl<T, U> Functor<T, U> for EnvScopeKind<T> {
type Target = EnvScopeKind<U>;
fn map(self, f: impl FnOnce(T) -> U) -> Self::Target {
use EnvScopeKind::*;
match self {
Shadow(x) => Shadow(f(x)),
Visible(x) => Visible(f(x)),
Hidden(x) => Hidden(f(x)),
}
}
}
impl<T> From<EnvScopeKind<T>> for Span
where
T: Into<Span>,
{
fn from(kind: EnvScopeKind<T>) -> Self {
kind.into_inner().into()
}
tamer: asg::air: Begin to introduce explicit scope testing There's a lot of documentation on this in the commit itself, but this stems from a) frustration with trying to understand how the system needs to operate with all of the objects involved; and b) recognizing that if I'm having difficulty, then others reading the system later on (including myself) and possibly looking to improve upon it are going to have a whole lot of trouble. Identifier scope is something I've been mulling over for years, and more formally for the past couple of months. This finally begins to formalize that, out of frustration with package imports. But it will be a weight lifted off of me as well, with issues of scope always looming. This demonstrates a declarative means of testing for scope by scanning the entire graph in tests to determine where an identifier has been scoped. Since no such scoping has been implemented yet, the tests demonstrate how they will look, but otherwise just test for current behavior. There is more existing behavior to check, and further there will be _references_ to check, as they'll also leave a trail of scope indexing behind as part of the resolution process. See the documentation introduced by this commit for more information on that part of this commit. Introducing the graph scanning, with the ASG's static assurances, required more lowering of dynamic types into the static types required by the API. This was itself a confusing challenge that, while not all that bad in retrospect, was something that I initially had some trouble with. The documentation includes clarifying remarks that hopefully make it all understandable. DEV-13162
2023-05-12 12:41:51 -04:00
}
impl AsMut<AirAggregateCtx> for AirAggregateCtx {
fn as_mut(&mut self) -> &mut AirAggregateCtx {
self
}
}
impl AsRef<Asg> for AirAggregateCtx {
fn as_ref(&self) -> &Asg {
&self.asg
}
}
impl AsMut<Asg> for AirAggregateCtx {
fn as_mut(&mut self) -> &mut Asg {
&mut self.asg
}
}
impl AsMut<AirStack> for AirAggregateCtx {
fn as_mut(&mut self) -> &mut AirStack {
&mut self.stack
}
}
impl From<Asg> for AirAggregateCtx {
fn from(asg: Asg) -> Self {
Self {
asg,
..Default::default()
}
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
#[cfg(test)]
pub mod test;