tame/tamer/src/asg/air.rs

315 lines
11 KiB
Rust
Raw Normal View History

tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
// ASG IR
//
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
//
// This file is part of TAME.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
//! Intermediate representation for construction of the
//! [abstract semantic graph (ASG)](super) (AIR).
//!
//! AIR serves as an abstraction layer between higher-level parsers and the
//! aggregate ASG.
//! It allows parsers to operate as a raw stream of data without having to
//! worry about ownership of or references to the ASG,
//! and allows for multiple such parsers to be joined.
//!
//! AIR is _not_ intended to replace the API of the ASG---it
//! is intended as a termination point for the parsing pipeline,
//! and as such implements a subset of the ASG's API that is suitable
//! for aggregating raw data from source and object files.
//! Given that it does so little and is so close to the [`Asg`] API,
//! one might say that the abstraction is as light as air,
//! but that would surely result in face-palming and so we're not going
//! air such cringeworthy dad jokes here.
use super::{
graph::object::Pkg,
Asg, AsgError, ObjectIndex,
};
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
use crate::{
diagnose::Annotate, diagnostic_unreachable, parse::prelude::*,
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
sym::SymbolId,
};
use std::fmt::{Debug, Display};
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
#[macro_use]
mod ir;
pub use ir::Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
mod expr;
mod tpl;
use expr::AirExprAggregate;
use tpl::AirTplAggregate;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
pub type IdentSym = SymbolId;
pub type DepSym = SymbolId;
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// AIR parser state.
#[derive(Debug, PartialEq, Default)]
pub enum AirAggregate {
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// Parser is not currently performing any work.
#[default]
Empty,
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
/// Expecting a package-level token.
PkgHead(ObjectIndex<Pkg>, AirExprAggregate<Pkg>),
/// Parsing an expression.
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
///
/// This expects to inherit an [`AirExprAggregate`] from the prior state
/// so that we are not continuously re-allocating its stack for each
/// new expression root.
PkgExpr(ObjectIndex<Pkg>, AirExprAggregate<Pkg>),
/// Parser is in template parsing mode.
///
/// All objects encountered until the closing [`Air::TplClose`] will be
/// parented to this template rather than the parent [`Pkg`].
/// See [`Air::TplOpen`] for more information.
PkgTpl(ObjectIndex<Pkg>, AirExprAggregate<Pkg>, AirTplAggregate),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
impl Display for AirAggregate {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
use AirAggregate::*;
match self {
Empty => write!(f, "awaiting AIR input for ASG"),
PkgHead(_, _) => {
write!(f, "expecting package header or an expression")
}
PkgExpr(_, expr) => {
write!(f, "defining a package expression: {expr}")
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
PkgTpl(_, _, tpl) => {
write!(f, "building a template: {tpl}",)
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
impl ParseState for AirAggregate {
type Token = Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
type Object = ();
type Error = AsgError;
/// Destination [`Asg`] that this parser lowers into.
///
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
/// This ASG will be yielded by [`crate::parse::Parser::finalize`].
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
type Context = Asg;
fn parse_token(
self,
tok: Self::Token,
asg: &mut Self::Context,
) -> crate::parse::TransitionResult<Self> {
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
use ir::{
AirIdent::*, AirPkg::*, AirSubsets::*, AirTodo::*, AirTpl::*,
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
};
use AirAggregate::*;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
// TODO: Seems to be about time for refactoring this...
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
match (self, tok.into()) {
(st, AirTodo(Todo(_))) => Transition(st).incomplete(),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
(Empty, AirPkg(PkgOpen(span))) => {
let oi_pkg = asg.create(Pkg::new(span)).root(asg);
Transition(PkgHead(oi_pkg, AirExprAggregate::new_in(oi_pkg)))
.incomplete()
}
(PkgHead(oi_pkg, expr), AirPkg(PkgOpen(span))) => {
Transition(PkgHead(oi_pkg, expr))
.err(AsgError::NestedPkgOpen(span, oi_pkg.span()))
}
(PkgExpr(oi_pkg, expr), AirPkg(PkgOpen(span))) => {
Transition(PkgExpr(oi_pkg, expr))
.err(AsgError::NestedPkgOpen(span, oi_pkg.span()))
}
// No expression was started.
(PkgHead(oi_pkg, _expr), AirPkg(PkgClose(span))) => {
oi_pkg.close(asg, span);
Transition(Empty).incomplete()
}
(PkgHead(..), AirBind(ident)) => {
todo!("PkgBody AirBind {ident:?}")
}
(PkgHead(oi_pkg, expr), tok @ AirExpr(..)) => {
Transition(PkgExpr(oi_pkg, expr))
.incomplete()
.with_lookahead(tok)
}
// Note that templates may preempt expressions at any point,
// unlike in NIR at the time of writing.
(
PkgHead(oi_pkg, expr) | PkgExpr(oi_pkg, expr),
tok @ AirTpl(..),
) => Transition(PkgTpl(
oi_pkg,
expr,
AirTplAggregate::new_in_pkg(oi_pkg),
))
.incomplete()
.with_lookahead(tok),
// Note: We unfortunately can't match on `AirExpr | AirBind`
// and delegate in the same block
// (without having to duplicate type checks and then handle
// unreachable paths)
// because of the different inner types.
(PkgExpr(oi_pkg, expr), AirExpr(etok)) => {
Self::delegate_expr(asg, oi_pkg, expr, etok)
}
(PkgExpr(oi_pkg, expr), AirBind(etok)) => {
Self::delegate_expr(asg, oi_pkg, expr, etok)
}
// Templates can contain just about anything,
// so completely hand over parsing when we're in template mode.
(PkgTpl(oi_pkg, stored_expr, tplst), tok) => {
Self::delegate_tpl(asg, oi_pkg, stored_expr, tplst, tok.into())
}
(Empty, AirTpl(TplClose(..))) => {
todo!("Empty AirTpl::TplClose")
}
(Empty, AirPkg(PkgClose(span))) => {
Transition(Empty).err(AsgError::InvalidPkgCloseContext(span))
}
(PkgExpr(oi_pkg, expr), AirPkg(PkgClose(span))) => {
match expr.is_accepting(asg) {
true => {
// TODO: this is duplicated with the above
oi_pkg.close(asg, span);
Transition(Empty).incomplete()
}
false => Transition(PkgExpr(oi_pkg, expr))
.err(AsgError::InvalidPkgCloseContext(span)),
}
}
(Empty, tok @ (AirExpr(..) | AirBind(..) | AirTpl(TplOpen(_)))) => {
Transition(Empty).err(AsgError::PkgExpected(tok.span()))
}
(Empty, AirIdent(IdentDecl(name, kind, src))) => {
asg.declare(name, kind, src).map(|_| ()).transition(Empty)
}
(Empty, AirIdent(IdentExternDecl(name, kind, src))) => asg
.declare_extern(name, kind, src)
.map(|_| ())
.transition(Empty),
(Empty, AirIdent(IdentDep(sym, dep))) => {
asg.add_dep_lookup(sym, dep);
Transition(Empty).incomplete()
}
(Empty, AirIdent(IdentFragment(sym, text))) => {
asg.set_fragment(sym, text).map(|_| ()).transition(Empty)
}
(Empty, AirIdent(IdentRoot(sym))) => {
let obj = asg.lookup_or_missing(sym);
asg.add_root(obj);
Transition(Empty).incomplete()
}
(st, tok @ AirIdent(..)) => todo!("{st:?}, {tok:?}"),
}
}
fn is_accepting(&self, _: &Self::Context) -> bool {
matches!(self, Self::Empty)
}
}
impl AirAggregate {
/// Delegate to the expression parser [`AirExprAggregate`].
///
/// TODO: This ought to be further reduced into primitives in the core
/// [`crate::parse`] framework.
fn delegate_expr(
asg: &mut <Self as ParseState>::Context,
oi_pkg: ObjectIndex<Pkg>,
expr: AirExprAggregate<Pkg>,
etok: impl Into<<AirExprAggregate<Pkg> as ParseState>::Token>,
) -> TransitionResult<Self> {
let tok = etok.into();
let tokspan = tok.span();
expr.parse_token(tok, asg).branch_dead::<Self, _>(
// TODO: Enforce using type system to avoid need for this
// runtime check and prove that it is indeed impossible
// (which otherwise could fail to be the case due to changes
// since this was written).
|_, ()| {
diagnostic_unreachable!(
vec![tokspan.internal_error(
"unexpected dead state transition at this token"
)],
"AirExprAggregate should not have dead states"
)
},
|expr, result, ()| {
result
.map(ParseStatus::reflexivity)
.transition(Self::PkgExpr(oi_pkg, expr))
},
(),
)
}
/// Delegate to the expression parser [`AirTplAggregate`].
///
/// After template parsing is complete
/// (when reaching a dead state),
/// the stored expression [`AirExprAggregate`] is reinstated,
/// allowing parsing to continue where it left off before being
/// preempted by template parsing.
fn delegate_tpl(
asg: &mut <Self as ParseState>::Context,
oi_pkg: ObjectIndex<Pkg>,
stored_expr: AirExprAggregate<Pkg>,
tplst: AirTplAggregate,
tok: Air,
) -> TransitionResult<Self> {
tplst.parse_token(tok, asg).branch_dead::<Self, _>(
|_, stored_expr| {
Transition(Self::PkgExpr(oi_pkg, stored_expr)).incomplete()
},
|tplst, result, stored_expr| {
result
.map(ParseStatus::reflexivity)
.transition(Self::PkgTpl(oi_pkg, stored_expr, tplst))
},
stored_expr,
)
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
#[cfg(test)]
mod test;