tame/tamer/src/asg/air.rs

763 lines
26 KiB
Rust
Raw Normal View History

tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
// ASG IR
//
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
//
// This file is part of TAME.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
//! Intermediate representation for construction of the
//! [abstract semantic graph (ASG)](super) (AIR).
//!
//! AIR serves as an abstraction layer between higher-level parsers and the
//! aggregate ASG.
//! It allows parsers to operate as a raw stream of data without having to
//! worry about ownership of or references to the ASG,
//! and allows for multiple such parsers to be joined.
//!
//! AIR is _not_ intended to replace the API of the ASG---it
//! is intended as a termination point for the parsing pipeline,
//! and as such implements a subset of the ASG's API that is suitable
//! for aggregating raw data from source and object files.
//! Given that it does so little and is so close to the [`Asg`] API,
//! one might say that the abstraction is as light as air,
//! but that would surely result in face-palming and so we're not going
//! air such cringeworthy dad jokes here.
use self::ir::AirBindableExpr;
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
use super::{
graph::object::{Expr, Pkg},
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
Asg, AsgError, ObjectIndex,
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
};
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
use crate::{
asg::graph::object::Tpl,
diagnose::Annotate,
diagnostic_unreachable,
tamer: f::Functor: New trait This commit is purposefully coupled with changes that utilize it to demonstrate that the need for this abstraction has been _derived_, not forced; TAMER doesn't aim to be functional for the sake of it, since idiomatic Rust achieves many of its benefits without the formalisms. But, the formalisms do occasionally help, and this is one such example. There is other existing code that can be refactored to take advantage of this style as well. I do _not_ wish to pull an existing functional dependency into TAMER; I want to keep these abstractions light, and eliminate them as necessary, as Rust continues to integrate new features into its core. I also want to be able to modify the abstractions to suit our particular needs. (This is _not_ a general recommendation; it's particular to TAMER and to my experience.) This implementation of `Functor` is one such example. While it is modeled after Haskell in that it provides `fmap`, the primitive here is instead `map`, with `fmap` derived from it, since `map` allows for better use of Rust idioms. Furthermore, it's polymorphic over _trait_ type parameters, not method, allowing for separate trait impls for different container types, which can in turn be inferred by Rust and allow for some very concise mapping; this is particularly important for TAMER because of the disciplined use of newtypes. For example, `foo.overwrite(span)` and `foo.overwrite(name)` are both self-documenting, and better alternatives than, say, `foo.map_span(|_| span)` and `foo.map_symbol(|_| name)`; the latter are perfectly clear in what they do, but lack a layer of abstraction, and are verbose. But the clarity of the _new_ form does rely on either good naming conventions of arguments, or explicit type annotations using turbofish notation if necessary. This will be implemented on core Rust types as appropriate and as possible. At the time of writing, we do not yet have trait specialization, and there's too many soundness issues for me to be comfortable enabling it, so that limits that we can do with something like, say, a generic `Result`, while also allowing for specialized implementations based on newtypes. DEV-13160
2023-01-04 12:30:18 -05:00
f::Functor,
fmt::{DisplayWrapper, TtQuote},
parse::{prelude::*, util::SPair},
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
span::Span,
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
sym::SymbolId,
};
use std::fmt::{Debug, Display};
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
#[macro_use]
mod ir;
pub use ir::Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
pub type IdentSym = SymbolId;
pub type DepSym = SymbolId;
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// Stack of held expressions,
/// with the root expression at the bottom of the stack.
///
/// Expression [`ObjectIndex`]es are pushed onto this stack when
/// parsing a subexpression,
/// and are popped when the subexpression terminates.
/// The active expression is _not_ stored on this stack to avoid unnecessary
/// indirection.
///
/// Despite the immutable interface,
/// this does modify the inner [`Vec`] in-place;
/// it does not reallocate unless its capacity has been reached.
///
/// Unlike other parts of the system,
/// this is heap-allocated,
/// but should be very cache-friendly.
/// This reason for heap allocation is that this is explicitly
/// _unbounded_—systems like code generators ought to be able to output
/// expressions in a tacit style without worrying about arbitrary limits.
/// It is worth noting that the other parts of the system using
/// stack-allocated data structures is less about performance and more
/// about the simplicity afforded by keeping allocators out of the picture.
/// We'll address performance issues if they appear during profiling.
///
/// Another benefit of using [`Vec`] here is that Rust is able to properly
/// optimize away `memcpy`s for it,
/// rather than having to utilize the parser's mutable context.
/// Further,
/// the ASG is heap-allocated,
/// so we're not avoiding the heap anyway.
///
/// The interface is modeled after [Haskell's `Stack`][haskell-stack],
/// with a slight variation for [`Self::pop`] so that we can avoid
/// reallocation after a stack is used up,
/// which is frequent.
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
///
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// [haskell-stack]: https://hackage.haskell.org/package/Stack/docs/Data-Stack.html
///
/// The stack states [`Dormant`] and [`Active`] selectively provide
/// different APIs to enforce certain invariants,
/// as an alternative to re-allocating an inner [`Vec`] each time a new
/// root expression is encountered.
#[derive(Debug, PartialEq, Eq)]
pub struct ExprStack<S>(Vec<ObjectIndex<Expr>>, S);
/// Expression stack is not in use and must be empty;
/// no ongoing expression parsing.
#[derive(Debug, PartialEq, Eq)]
pub struct Dormant;
/// Expression stack is in use as part of an expression parse.
#[derive(Debug, PartialEq, Eq)]
pub struct Active(StackEdge);
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
#[derive(Debug, PartialEq, Eq)]
pub enum StackEdge {
/// Root expression is yet not reachable from any other object.
///
/// Dangling expressions are expected to transition into
/// [`Self::Reachable`] after being bound to an identifier.
/// Closing a dangling expression will result in a
/// [`AsgError::DanglingExpr`].
///
/// Binding a sub-expression does not bind the root of the stack,
/// since sub-expressions cannot reference their parent;
/// a stack is dangling until its root expression has been bound to
/// an identifier.
Dangling,
/// Root expression is reachable from another object.
///
/// The associated [`SPair`] serves as _evidence_ of this assertion.
Reachable(SPair),
}
impl Display for StackEdge {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
match self {
Self::Dangling => write!(f, "dangling"),
Self::Reachable(ident) => {
write!(f, "reachable (by {})", TtQuote::wrap(ident))
}
}
}
}
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
impl ExprStack<Dormant> {
/// Mark the stack as active,
/// exposing its stack API for use.
///
/// [`ExprStack::done`] will return the stack to a dormant state.
fn activate(self) -> ExprStack<Active> {
let Self(stack, _) = self;
ExprStack(stack, Active(StackEdge::Dangling))
}
}
impl ExprStack<Active> {
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
fn push(self, item: ObjectIndex<Expr>) -> Self {
let Self(mut stack, s) = self;
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
stack.push(item);
Self(stack, s)
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
/// Attempt to remove an item from the stack,
/// returning a new stack and the item,
/// if any.
///
/// This returns a new [`Self`] even if it is empty so that it can be
/// reused without having to reallocate.
fn pop(self) -> (Self, Option<ObjectIndex<Expr>>) {
let Self(mut stack, s) = self;
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
let oi = stack.pop();
(Self(stack, s), oi)
}
/// Whether the stack is dangling.
fn is_dangling(&self) -> bool {
matches!(self, Self(_, Active(StackEdge::Dangling)))
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
/// Mark stack as reachable if processing the root expression.
///
/// `ident` is admitted as evidence of reachability,
/// both for debugging and for making it more difficult to
/// misuse this API.
/// If the stack is already reachable,
/// the previous identifier takes precedence.
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
///
/// If not parsing the root expression
/// (if the stack is non-empty),
/// this returns `self` unchanged.
fn reachable_by(self, ident: SPair) -> Self {
match self {
Self(stack, Active(StackEdge::Dangling)) if stack.is_empty() => {
Self(stack, Active(StackEdge::Reachable(ident)))
}
_ => self,
}
}
/// Mark the stack as dormant,
/// hiding its stack API and ensuring that its state is properly reset
/// for the next root expression.
///
/// [`ExprStack::activate`] will re-activate the stack for use.
fn done(self) -> ExprStack<Dormant> {
let Self(stack, _) = self;
// TODO: error if non-empty stack (unclosed expr)
if !stack.is_empty() {
todo!("ExprStack::done(): error on non-empty stack")
}
ExprStack(stack, Dormant)
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
}
impl Default for ExprStack<Dormant> {
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
fn default() -> Self {
// TODO: 16 is a generous guess that is very unlikely to be exceeded
// in practice at the time of writing,
// even with template expansion,
// but let's develop an informed heuristic.
// Note that this is very unlikely to make a difference;
// I just don't like using numbers without data to back them up.
Self(Vec::with_capacity(16), Dormant)
}
}
impl Display for ExprStack<Dormant> {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
let Self(stack, _) = self;
write!(f, "dormant expression stack of size {}", stack.capacity())
}
}
impl Display for ExprStack<Active> {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
let Self(stack, Active(edge_st)) = self;
write!(
f,
"active {edge_st} expression stack of length {} and size {}",
stack.len(),
stack.capacity()
)
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
}
/// AIR parser state.
#[derive(Debug, PartialEq, Eq, Default)]
pub enum AirAggregate {
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
/// Parser is not currently performing any work.
#[default]
Empty,
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
/// Expecting a package-level token.
PkgHead(ObjectIndex<Pkg>, AirExprAggregate),
/// Parsing an expression.
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
///
/// This expects to inherit an [`AirExprAggregate`] from the prior state
/// so that we are not continuously re-allocating its stack for each
/// new expression root.
PkgExpr(ObjectIndex<Pkg>, AirExprAggregate),
/// Parser is in template parsing mode.
///
/// All objects encountered until the closing [`Air::TplClose`] will be
/// parented to this template rather than the parent [`Pkg`].
/// See [`Air::TplOpen`] for more information.
PkgTpl(ObjectIndex<Pkg>, AirExprAggregate, AirTplAggregate),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
impl Display for AirAggregate {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
use AirAggregate::*;
match self {
Empty => write!(f, "awaiting AIR input for ASG"),
PkgHead(_, _) => {
write!(f, "expecting package header or an expression")
}
PkgExpr(_, expr) => {
write!(f, "defining a package expression: {expr}")
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
PkgTpl(_, _, tpl) => {
write!(f, "building a template: {tpl}",)
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
impl ParseState for AirAggregate {
type Token = Air;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
type Object = ();
type Error = AsgError;
/// Destination [`Asg`] that this parser lowers into.
///
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
/// This ASG will be yielded by [`crate::parse::Parser::finalize`].
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
type Context = Asg;
fn parse_token(
self,
tok: Self::Token,
asg: &mut Self::Context,
) -> crate::parse::TransitionResult<Self> {
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
use ir::{
AirIdent::*, AirPkg::*, AirSubsets::*, AirTodo::*, AirTpl::*,
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
};
use AirAggregate::*;
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
// TODO: Seems to be about time for refactoring this...
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
match (self, tok.into()) {
(st, AirTodo(Todo(_))) => Transition(st).incomplete(),
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
(Empty, AirPkg(PkgOpen(span))) => {
let oi_pkg = asg.create(Pkg::new(span)).root(asg);
Transition(PkgHead(oi_pkg, AirExprAggregate::parse_pkg(oi_pkg)))
.incomplete()
}
(PkgHead(oi_pkg, expr), AirPkg(PkgOpen(span))) => {
Transition(PkgHead(oi_pkg, expr))
.err(AsgError::NestedPkgOpen(span, oi_pkg.span()))
}
(PkgExpr(oi_pkg, expr), AirPkg(PkgOpen(span))) => {
Transition(PkgExpr(oi_pkg, expr))
.err(AsgError::NestedPkgOpen(span, oi_pkg.span()))
}
// No expression was started.
(PkgHead(oi_pkg, _expr), AirPkg(PkgClose(span))) => {
oi_pkg.close(asg, span);
Transition(Empty).incomplete()
}
(PkgHead(..), AirBind(ident)) => {
todo!("PkgBody AirBind {ident:?}")
}
(PkgHead(oi_pkg, expr), tok @ AirExpr(..)) => {
Transition(PkgExpr(oi_pkg, expr))
.incomplete()
.with_lookahead(tok)
}
// Note that templates may preempt expressions at any point,
// unlike in NIR at the time of writing.
(
PkgHead(oi_pkg, expr) | PkgExpr(oi_pkg, expr),
tok @ AirTpl(..),
) => Transition(PkgTpl(
oi_pkg,
expr,
AirTplAggregate::define_in_pkg(oi_pkg),
))
.incomplete()
.with_lookahead(tok),
// Note: We unfortunately can't match on `AirExpr | AirBind`
// and delegate in the same block
// (without having to duplicate type checks and then handle
// unreachable paths)
// because of the different inner types.
(PkgExpr(oi_pkg, expr), AirExpr(etok)) => {
Self::delegate_expr(asg, oi_pkg, expr, etok)
}
(PkgExpr(oi_pkg, expr), AirBind(etok)) => {
Self::delegate_expr(asg, oi_pkg, expr, etok)
}
// Templates can contain just about anything,
// so completely hand over parsing when we're in template mode.
(PkgTpl(oi_pkg, stored_expr, tplst), tok) => {
Self::delegate_tpl(asg, oi_pkg, stored_expr, tplst, tok.into())
}
(Empty, AirTpl(TplClose(..))) => {
todo!("Empty AirTpl::TplClose")
}
(Empty, AirPkg(PkgClose(span))) => {
Transition(Empty).err(AsgError::InvalidPkgCloseContext(span))
}
(PkgExpr(oi_pkg, expr), AirPkg(PkgClose(span))) => {
match expr.is_accepting(asg) {
true => {
// TODO: this is duplicated with the above
oi_pkg.close(asg, span);
Transition(Empty).incomplete()
}
false => Transition(PkgExpr(oi_pkg, expr))
.err(AsgError::InvalidPkgCloseContext(span)),
}
}
(Empty, tok @ (AirExpr(..) | AirBind(..) | AirTpl(TplOpen(_)))) => {
Transition(Empty).err(AsgError::PkgExpected(tok.span()))
}
(Empty, AirIdent(IdentDecl(name, kind, src))) => {
asg.declare(name, kind, src).map(|_| ()).transition(Empty)
}
(Empty, AirIdent(IdentExternDecl(name, kind, src))) => asg
.declare_extern(name, kind, src)
.map(|_| ())
.transition(Empty),
(Empty, AirIdent(IdentDep(sym, dep))) => {
asg.add_dep_lookup(sym, dep);
Transition(Empty).incomplete()
}
(Empty, AirIdent(IdentFragment(sym, text))) => {
asg.set_fragment(sym, text).map(|_| ()).transition(Empty)
}
(Empty, AirIdent(IdentRoot(sym))) => {
let obj = asg.lookup_or_missing(sym);
asg.add_root(obj);
Transition(Empty).incomplete()
}
(st, tok @ AirIdent(..)) => todo!("{st:?}, {tok:?}"),
}
}
fn is_accepting(&self, _: &Self::Context) -> bool {
matches!(self, Self::Empty)
}
}
impl AirAggregate {
/// Delegate to the expression parser [`AirExprAggregate`].
///
/// TODO: This ought to be further reduced into primitives in the core
/// [`crate::parse`] framework.
fn delegate_expr(
asg: &mut <Self as ParseState>::Context,
oi_pkg: ObjectIndex<Pkg>,
expr: AirExprAggregate,
etok: impl Into<<AirExprAggregate as ParseState>::Token>,
) -> TransitionResult<Self> {
let tok = etok.into();
let tokspan = tok.span();
expr.parse_token(tok, asg).branch_dead::<Self, _>(
// TODO: Enforce using type system to avoid need for this
// runtime check and prove that it is indeed impossible
// (which otherwise could fail to be the case due to changes
// since this was written).
|_, ()| {
diagnostic_unreachable!(
vec![tokspan.internal_error(
"unexpected dead state transition at this token"
)],
"AirExprAggregate should not have dead states"
)
},
|expr, result, ()| {
result
.map(ParseStatus::reflexivity)
.transition(Self::PkgExpr(oi_pkg, expr))
},
(),
)
}
/// Delegate to the expression parser [`AirTplAggregate`].
///
/// After template parsing is complete
/// (when reaching a dead state),
/// the stored expression [`AirExprAggregate`] is reinstated,
/// allowing parsing to continue where it left off before being
/// preempted by template parsing.
fn delegate_tpl(
asg: &mut <Self as ParseState>::Context,
oi_pkg: ObjectIndex<Pkg>,
stored_expr: AirExprAggregate,
tplst: AirTplAggregate,
tok: Air,
) -> TransitionResult<Self> {
tplst
.parse_token(tok, asg)
.branch_dead::<Self, AirExprAggregate>(
|_, stored_expr| {
Transition(Self::PkgExpr(oi_pkg, stored_expr)).incomplete()
},
|tplst, result, stored_expr| {
result
.map(ParseStatus::reflexivity)
.transition(Self::PkgTpl(oi_pkg, stored_expr, tplst))
},
stored_expr,
)
}
}
/// Parse an AIR expression with binding support.
///
/// Expressions are composable,
/// so this parser need only care about whether it has any active
/// expression being parsed.
///
/// This parser has no dead states---it
/// handles each of its tokens and performs error recovery on invalid
/// state transitions.
#[derive(Debug, PartialEq, Eq)]
pub enum AirExprAggregate {
/// Ready for an expression;
/// expression stack is empty.
Ready(ObjectIndex<Pkg>, ExprStack<Dormant>),
/// Building an expression.
BuildingExpr(ObjectIndex<Pkg>, ExprStack<Active>, ObjectIndex<Expr>),
}
impl Display for AirExprAggregate {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
match self {
Self::Ready(_, es) => write!(f, "ready for expression with {es}"),
Self::BuildingExpr(_, es, _) => {
write!(f, "building expression with {es}")
}
}
}
}
impl ParseState for AirExprAggregate {
type Token = AirBindableExpr;
type Object = ();
type Error = AsgError;
type Context = Asg;
fn parse_token(
self,
tok: Self::Token,
asg: &mut Self::Context,
) -> crate::parse::TransitionResult<Self::Super> {
use ir::{AirBind::*, AirExpr::*};
use AirBindableExpr::*;
use AirExprAggregate::*;
match (self, tok) {
(Ready(oi_pkg, es), AirExpr(ExprOpen(op, span))) => {
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
let oi = asg.create(Expr::new(op, span));
Transition(BuildingExpr(oi_pkg, es.activate(), oi)).incomplete()
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
(BuildingExpr(oi_pkg, es, poi), AirExpr(ExprOpen(op, span))) => {
let oi = poi.create_subexpr(asg, Expr::new(op, span));
Transition(BuildingExpr(oi_pkg, es.push(poi), oi)).incomplete()
}
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
(BuildingExpr(oi_pkg, es, oi), AirExpr(ExprClose(end))) => {
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
let start: Span = oi.into();
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
let _ = oi.map_obj(asg, |expr| {
tamer: f::Functor: New trait This commit is purposefully coupled with changes that utilize it to demonstrate that the need for this abstraction has been _derived_, not forced; TAMER doesn't aim to be functional for the sake of it, since idiomatic Rust achieves many of its benefits without the formalisms. But, the formalisms do occasionally help, and this is one such example. There is other existing code that can be refactored to take advantage of this style as well. I do _not_ wish to pull an existing functional dependency into TAMER; I want to keep these abstractions light, and eliminate them as necessary, as Rust continues to integrate new features into its core. I also want to be able to modify the abstractions to suit our particular needs. (This is _not_ a general recommendation; it's particular to TAMER and to my experience.) This implementation of `Functor` is one such example. While it is modeled after Haskell in that it provides `fmap`, the primitive here is instead `map`, with `fmap` derived from it, since `map` allows for better use of Rust idioms. Furthermore, it's polymorphic over _trait_ type parameters, not method, allowing for separate trait impls for different container types, which can in turn be inferred by Rust and allow for some very concise mapping; this is particularly important for TAMER because of the disciplined use of newtypes. For example, `foo.overwrite(span)` and `foo.overwrite(name)` are both self-documenting, and better alternatives than, say, `foo.map_span(|_| span)` and `foo.map_symbol(|_| name)`; the latter are perfectly clear in what they do, but lack a layer of abstraction, and are verbose. But the clarity of the _new_ form does rely on either good naming conventions of arguments, or explicit type annotations using turbofish notation if necessary. This will be implemented on core Rust types as appropriate and as possible. At the time of writing, we do not yet have trait specialization, and there's too many soundness issues for me to be comfortable enabling it, so that limits that we can do with something like, say, a generic `Result`, while also allowing for specialized implementations based on newtypes. DEV-13160
2023-01-04 12:30:18 -05:00
expr.map(|span| span.merge(end).unwrap_or(span))
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
});
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
match es.pop() {
(es, Some(poi)) => {
Transition(BuildingExpr(oi_pkg, es, poi)).incomplete()
}
(es, None) => {
let dangling = es.is_dangling();
let st = Ready(oi_pkg, es.done());
if dangling {
Transition(st).err(AsgError::DanglingExpr(
start.merge(end).unwrap_or(start),
))
} else {
Transition(st).incomplete()
}
}
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
}
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
tamer: asg::air: AIR as a sum IR This introduces a new macro `sum_ir!` to help with a long-standing problem of not being able to easily narrow types in Rust without a whole lot of boilerplate. This patch includes a bit of documentation, so see that for more information. This was not a welcome change---I jumped down this rabbit hole trying to decompose `AirAggregate` so that I can share portions of parsing with the current parser and a template parser. I can now proceed with that. This is not the only implementation that I had tried. I previously inverted the approach, as I've been doing manually for some time: manually create types to hold the sets of variants, and then create a sum type to hold those types. That works, but it resulted in a mess for systems that have to use the IR, since now you have two enums to contend with. I didn't find that to be appropriate, because we shouldn't complicate the external API for implementation details. The enum for IRs is supposed to be like a bytecode---a list of operations that can be performed with the IR. They can be grouped if it makes sense for a public API, but in my case, I only wanted subsets for the sake of delegating responsibilities to smaller subsystems, while retaining the context that `match` provides via its exhaustiveness checking but does not expose as something concrete (which is deeply frustrating!). Anyway, here we are; this'll be refined over time, hopefully, and portions of it can be generalized for removing boilerplate from other IRs. Another thing to note is that this syntax is really a compromise---I had to move on, and I was spending too much time trying to get creative with `macro_rules!`. It isn't the best, and it doesn't seem very Rust-like in some places and is therefore not necessarily all that intuitive. This can be refined further in the future. But the end result, all things considered, isn't too bad. DEV-13708
2023-03-02 15:15:28 -05:00
(BuildingExpr(oi_pkg, es, oi), AirBind(BindIdent(id))) => {
let oi_ident = asg.lookup_or_missing(id);
oi_pkg.defines(asg, oi_ident);
tamer: asg::air::AirAggregate: Initial impl of nested exprs This introduces a number of concepts together, again to demonstrate that they were derived. This introduces support for nested expressions, extending the previous work. It also supports error recovery for dangling expressions. The parser states are a mess; there is a lot of duplicate code here that needs refactoring, but I wanted to commit this first at a known-good state so that the diff will demonstrate the need for the change that will follow; the opportunities for abstraction are plainly visible. The immutable stack introduced here could be generalized, if needed, in the future. Another important note is that Rust optimizes away the `memcpy`s for the stack that was introduced here. The initial Parser Context was introduced because of `ArrayVec` inhibiting that elision, but Vec never had that problem. In the future, I may choose to go back and remove ArrayVec, but I had wanted to keep memory allocation out of the picture as much as possible to make the disassembly and call graph easier to reason about and to have confidence that optimizations were being performed as intended. With that said---it _should_ be eliding in tamec, since we're not doing anything meaningful yet with the graph. It does also elide in tameld, but it's possible that Rust recognizes that those code paths are never taken because tameld does nothing with expressions. So I'll have to monitor this as I progress and adjust accordingly; it's possible a future commit will call BS on everything I just said. Of course, the counter-point to that is that Rust is optimizing them away anyway, but Vec _does_ still require allocation; I was hoping to keep such allocation at the fringes. But another counter-point is that it _still_ is allocated at the fringe, when the context is initialized for the parser as part of the lowering pipeline. But I didn't know how that would all come together back then. ...alright, enough rambling. DEV-13160
2023-01-05 15:57:06 -05:00
// It is important that we do not mark this expression as
// reachable unless we successfully bind the identifier.
match oi_ident.bind_definition(asg, id, oi) {
Ok(_) => Transition(BuildingExpr(
oi_pkg,
es.reachable_by(id),
oi,
))
.incomplete(),
Err(e) => Transition(BuildingExpr(oi_pkg, es, oi)).err(e),
}
tamer: Initial concept for AIR/ASG Expr This begins to place expressions on the graph---something that I've been thinking about for a couple of years now, so it's interesting to finally be doing it. This is going to evolve; I want to get some things committed so that it's clear how I'm moving forward. The ASG makes things a bit awkward for a number of reasons: 1. I'm dealing with older code where I had a different model of doing things; 2. It's mutable, rather than the mostly-functional lowering pipeline; 3. We're dealing with an aggregate ever-evolving blob of data (the graph) rather than a stream of tokens; and 4. We don't have as many type guarantees. I've shown with the lowering pipeline that I'm able to take a mutable reference and convert it into something that's both functional and performant, where I remove it from its container (an `Option`), create a new version of it, and place it back. Rust is able to optimize away the memcpys and such and just directly manipulate the underlying value, which is often a register with all of the inlining. _But_ this is a different scenario now. The lowering pipeline has a narrow context. The graph has to keep hitting memory. So we'll see how this goes. But it's most important to get this working and measure how it performs; I'm not trying to prematurely optimize. My attempts right now are for the way that I wish to develop. Speaking to #4 above, it also sucks that I'm not able to type the relationships between nodes on the graph. Rather, it's not that I _can't_, but a project to created a typed graph library is beyond the scope of this work and would take far too much time. I'll leave that to a personal, non-work project. Instead, I'm going to have to narrow the type any time the graph is accessed. And while that sucks, I'm going to do my best to encapsulate those details to make it as seamless as possible API-wise. The performance hit of performing the narrowing I'm hoping will be very small relative to all the business logic going on (a single cache miss is bound to be far more expensive than many narrowings which are just integer comparisons and branching)...but we'll see. Introducing branching sucks, but branch prediction is pretty damn good in modern CPUs. DEV-13160
2022-12-21 16:47:04 -05:00
}
(BuildingExpr(oi_pkg, es, oi), AirBind(RefIdent(ident))) => {
Transition(BuildingExpr(oi_pkg, es, oi.ref_expr(asg, ident)))
.incomplete()
}
(st @ Ready(..), AirBind(BindIdent(id))) => {
Transition(st).err(AsgError::InvalidExprBindContext(id))
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
(st @ Ready(..), AirBind(RefIdent(id))) => {
Transition(st).err(AsgError::InvalidExprRefContext(id))
}
(st @ Ready(..), AirExpr(ExprClose(span))) => {
Transition(st).err(AsgError::UnbalancedExpr(span))
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
}
fn is_accepting(&self, _: &Self::Context) -> bool {
matches!(self, Self::Ready(..))
}
}
impl AirExprAggregate {
fn parse_pkg(oi: ObjectIndex<Pkg>) -> Self {
Self::Ready(oi, ExprStack::default())
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
}
/// Template parser and token aggregator.
///
/// A template consists of
///
/// - Metadata about the template,
/// including its parameters; and
/// - A collection of [`Air`] tokens representing the body of the
/// template that will be expanded into the application site when the
/// template is applied.
///
/// This contains an embedded [`AirExprAggregate`] parser for handling
/// expressions just the same as [`AirAggregate`] does with packages.
#[derive(Debug, PartialEq, Eq)]
pub enum AirTplAggregate {
/// Ready for a template,
/// defined as part of the given package.
///
/// This state also include the template header;
/// unlike NIR,
/// AIR has no restrictions on when template header tokens are
/// provided,
/// which simplifies AIR generation.
///
/// The expression parser [`AirExprAggregate`] is initialized at this
/// state to avoid re-allocating its stack for adjacent templates.
/// This will not save any allocations if,
/// after reaching a dead state,
/// the caller
/// (or parent parser)
/// chooses to deallocate us.
Ready(ObjectIndex<Pkg>, AirExprAggregate),
/// Aggregating tokens into a template.
BuildingTpl(
ObjectIndex<Pkg>,
ObjectIndex<Tpl>,
AirExprAggregate,
Option<SPair>,
),
}
impl Display for AirTplAggregate {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
match self {
Self::Ready(_, _) => write!(f, "ready for template definition"),
Self::BuildingTpl(_, expr, _, None) => {
write!(f, "building anonymous template with {expr}")
}
Self::BuildingTpl(_, expr, _, Some(name)) => {
write!(
f,
"building named template {} with {expr}",
TtQuote::wrap(name)
)
}
}
}
}
impl ParseState for AirTplAggregate {
type Token = Air;
type Object = ();
type Error = AsgError;
type Context = Asg;
fn parse_token(
self,
tok: Self::Token,
asg: &mut Self::Context,
) -> TransitionResult<Self::Super> {
use ir::{AirBind::*, AirSubsets::*, AirTodo::*, AirTpl::*};
use AirTplAggregate::*;
match (self, tok.into()) {
(st, AirTodo(Todo(_))) => Transition(st).incomplete(),
(Ready(oi_pkg, expr), AirTpl(TplOpen(span))) => {
let oi_tpl = asg.create(Tpl::new(span));
Transition(BuildingTpl(oi_pkg, oi_tpl, expr, None)).incomplete()
}
(
BuildingTpl(oi_pkg, oi_tpl, expr, _),
AirBind(BindIdent(name)),
) => asg
.lookup_or_missing(name)
.bind_definition(asg, name, oi_tpl)
.map(|oi_ident| oi_pkg.defines(asg, oi_ident))
.map(|_| ())
.transition(BuildingTpl(oi_pkg, oi_tpl, expr, Some(name))),
(BuildingTpl(oi_pkg, oi_tpl, expr, _), AirTpl(TplClose(span))) => {
oi_tpl.close(asg, span);
Transition(Ready(oi_pkg, expr)).incomplete()
}
(BuildingTpl(..), AirPkg(_)) => {
todo!("template cannot define packages")
}
(BuildingTpl(..), tok) => todo!("BuildingTpl body: {tok:?}"),
(st @ Ready(..), AirTpl(TplClose(span))) => {
Transition(st).err(AsgError::UnbalancedTpl(span))
}
(
st @ Ready(..),
tok @ (AirPkg(..) | AirExpr(..) | AirBind(..) | AirIdent(..)),
) => Transition(st).dead(tok.into()),
}
}
fn is_accepting(&self, _: &Self::Context) -> bool {
matches!(self, Self::Ready(..))
}
}
impl AirTplAggregate {
fn define_in_pkg(oi_pkg: ObjectIndex<Pkg>) -> Self {
Self::Ready(oi_pkg, AirExprAggregate::parse_pkg(oi_pkg))
}
}
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
#[cfg(test)]
mod test;