tame/tamer/src/xir/attr.rs

261 lines
7.0 KiB
Rust
Raw Normal View History

// XIRT attributes
//
// Copyright (C) 2014-2022 Ryan Specialty Group, LLC.
//
// This file is part of TAME.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! XIRT attributes.
//!
//! Attributes are represented by [`Attr`].
//!
//! See [parent module](super) for additional documentation.
tamer: xir::tree::attr_parser_from: Integrate AttrParser This begins to integrate the isolated AttrParser. The next step will be integrating it into the larger XIRT parser. There's been considerable delay in getting this committed, because I went through quite the struggle with myself trying to determine what balance I want to strike between Rust's type system; convenience with parser combinators; iterators; and various other abstractions. I ended up being confounded by trying to maintain the current XmloReader abstraction, which is fundamentally incompatible with the way the new parsing system works (streaming iterators that do not collect or perform heap allocations). There'll be more information on this to come, but there are certain things that will be changing. There are a couple problems highlighted by this commit (not in code, but conceptually): 1. Introducing Option here for the TokenParserState doesn't feel right, in the sense that the abstraction is inappropriate. We should perhaps introduce a new variant Parsed::Done or something to indicate intent, rather than leaving the reader to have to read about what None actually means. 2. This turns Parsed into more of a statement influencing control flow/logic, and so should be encapsulated, with an external equivalent of Parsed that omits variants that ought to remain encapsulated. 3. TokenStreamState is true, but these really are the actual parsers; TokenStreamParser is more of a coordinator, and helps to abstract away some of the common logic so lower-level parsers do not have to worry about it. But calling it TokenStreamState is both a bit confusing and is an understatement---it _does_ hold the state, but it also holds the current parsing stack in its variants. Another thing that is not yet entirely clear is whether this AttrParser ought to care about detection of duplicate attributes, or if that should be done in a separate parser, perhaps even at the XIR level. The same can be said for checking for balanced tags. By pushing it to TokenStream in XIR, we would get a guaranteed check regardless of what parsers are used, which is attractive because it reduces the (almost certain-to-otherwise-occur) risk that individual parsers will not sufficiently check for semantically valid XML. But it does _potentially_ match error recovery more complicated. But at the same time, perhaps more specific parsers ought not care about recovery at that level. Anyway, point being, more to come, but I am disappointed how much time I'm spending considering parsing, given that there are so many things I need to move onto. I just want this done right and in a way that feels like it's working well with Rust while it's all in working memory, otherwise it's going to be a significant effort to get back into. DEV-11268
2021-12-10 14:13:02 -05:00
mod parse;
use super::QName;
use crate::{
parse::Token,
span::{Span, SpanLenSize},
sym::SymbolId,
};
2021-11-23 13:05:10 -05:00
use std::fmt::Display;
pub use parse::{AttrParseError, AttrParseState};
tamer: xir:tree: Begin work on composable XIRT parser The XIRT parser was initially written for test cases, so that unit tests should assert more easily on generated token streams (XIR). While it was planned, it wasn't clear what the eventual needs would be, which were expected to differ. Indeed, loading everything into a generic tree representation in memory is not appropriate---we should prefer streaming and avoiding heap allocations when they’re not necessary, and we should parse into an IR rather than a generic format, which ensures that the data follow a proper grammar and are semantically valid. When parsing attributes in an isolated context became necessary for the aforementioned task, the state machine of the XIRT parser was modified to accommodate. The opposite approach should have been taken---instead of adding complexity and special cases to the parser, and from a complex parser extracting a simple one (an attribute parser), we should be composing the larger (full XIRT) parser from smaller ones (e.g. attribute, child elements). A combinator, when used in a functional sense, refers not to combinatory logic but to the composition of more complex systems from smaller ones. The changes made as part of this commit begin to work toward combinators, though it's not necessarily evident yet (to you, the reader) how that'll work, since the code for it hasn't yet been written; this is commit is simply getting my work thusfar introduced so I can do some light refactoring before continuing on it. TAMER does not aim to introduce a parser combinator framework in its usual sense---it favors, instead, striking a proper balance with Rust’s type system that permits the convenience of combinators only in situations where they are needed, to avoid having to write new parser boilerplate. Specifically: 1. Rust’s type system should be used as combinators, so that parsers are automatically constructed from the type definition. 2. Primitive parsers are written as explicit automata, not as primitive combinators. 3. Parsing should directly produce IRs as a lowering operation below XIRT, rather than producing XIRT itself. That is, target IRs should consume XIRT and produce parse themselves immediately, during streaming. In the future, if more combinators are needed, they will be added; maybe this will eventually evolve into a more generic parser combinator framework for TAME, but that is certainly a waste of time right now. And, to be honest, I’m hoping that won’t be necessary.
2021-12-06 11:26:53 -05:00
/// Element attribute.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Attr(pub QName, pub SymbolId, pub AttrSpan);
/// Spans associated with attribute key and value.
///
/// The diagram below illustrates the behavior of `AttrSpan`.
/// Note that the extra spaces surrounding the `=` are intentional to
/// illustrate what the behavior ought to be.
/// Spans are represented by `|---|` intervals,
/// with the byte offset at each end,
/// and the single-letter span name centered below the interval.
/// `+` represents intersecting `-` and `|` lines.
///
/// ```text
/// <foo bar = "baz" />
/// |-| |+-+|
/// 5 7 13| |17
/// |K |Q|
/// | 14 16
/// | V |
/// |-----------|
/// A
/// ```
///
/// Above we have
///
/// - `A` = [`AttrSpan::span`];
/// - `K` = [`AttrSpan::key_span`];
/// - `V` = [`AttrSpan::value_span`]; and
/// - `Q` = [`AttrSpan::value_span_with_quotes`].
///
/// Note that this object assumes that the key and value span are adjacent
/// to one-another in the same [`span::Context`](crate::span::Context).
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct AttrSpan(pub Span, pub Span);
impl AttrSpan {
/// A [`Span`] covering the entire attribute token,
/// including the key,
/// _quoted_ value,
/// and everything in-between.
pub fn span(&self) -> Span {
let AttrSpan(k, _) = self;
// TODO: Move much of this into `Span`.
k.context().span(
k.offset(),
self.value_span_with_quotes()
.endpoints_saturated()
.1
.offset()
.saturating_sub(k.offset())
.try_into()
.unwrap_or(SpanLenSize::MAX),
)
}
/// The span associated with the name of the key.
///
/// This does _not_ include the following `=` or any surrounding
/// whitespace.
pub fn key_span(&self) -> Span {
let AttrSpan(k, _) = self;
*k
}
/// The span associated with the string value _inside_ the quotes,
/// not including the quotes themselves.
///
/// See [`AttrSpan`]'s documentation for an example.
pub fn value_span(&self) -> Span {
let AttrSpan(_, v) = self;
*v
}
/// The span associated with the string value _including_ the
/// surrounding quotes.
///
/// See [`AttrSpan`]'s documentation for an example.
pub fn value_span_with_quotes(&self) -> Span {
let AttrSpan(_, v) = self;
v.context()
.span(v.offset().saturating_sub(1), v.len().saturating_add(2))
}
}
impl Attr {
/// Construct a new simple attribute with a name, value, and respective
/// [`Span`]s.
#[inline]
pub fn new(name: QName, value: SymbolId, span: (Span, Span)) -> Self {
Self(name, value, AttrSpan(span.0, span.1))
}
/// Attribute name.
#[inline]
pub fn name(&self) -> QName {
self.0
}
/// Retrieve the value from the attribute.
///
/// Since [`SymbolId`] implements [`Copy`],
/// this returns an owned value.
#[inline]
pub fn value(&self) -> SymbolId {
self.1
}
}
impl Token for Attr {
fn span(&self) -> Span {
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
match self {
Attr(.., attr_span) => attr_span.span(),
tamer: Refactor asg_builder into obj::xmlo::lower and asg::air This finally uses `parse` all the way up to aggregation into the ASG, as can be seen by the mess in `poc`. This will be further simplified---I just need to get this committed so that I can mentally get it off my plate. I've been separating this commit into smaller commits, but there's a point where it's just not worth the effort anymore. I don't like making large changes such as this one. There is still work to do here. First, it's worth re-mentioning that `poc` means "proof-of-concept", and represents things that still need a proper home/abstraction. Secondly, `poc` is retrieving the context of two parsers---`LowerContext` and `Asg`. The latter is desirable, since it's the final aggregation point, but the former needs to be eliminated; in particular, packages need to be worked into the ASG so that `found` can be removed. Recursively loading `xmlo` files still happens in `poc`, but the compiler will need this as well. Once packages are on the ASG, along with their state, that responsibility can be generalized as well. That will then simplify lowering even further, to the point where hopefully everything has the same shape (once final aggregation has an abstraction), after which we can then create a final abstraction to concisely stitch everything together. Right now, Rust isn't able to infer `S` for `Lower<S, LS>`, which is unfortunate, but we'll be able to help it along with a more explicit abstraction. DEV-11864
2022-05-27 13:51:29 -04:00
}
}
}
impl crate::parse::Object for Attr {}
2021-11-23 13:05:10 -05:00
impl Display for Attr {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "`@{}=\"{}\"` at {}", self.0, self.1, self.2 .0)
2021-11-23 13:05:10 -05:00
}
}
/// List of attributes.
///
/// Attributes are ordered in XIR so that this IR will be suitable for code
/// formatters and linters.
///
/// This abstraction will allow us to manipulate the internal data so that
/// it is suitable for a particular task in the future
/// (e.g. O(1) lookups by attribute name).
#[derive(Debug, Clone, Eq, PartialEq, Default)]
pub struct AttrList {
attrs: Vec<Attr>,
}
impl AttrList {
/// Construct a new, empty attribute list.
pub fn new() -> Self {
Self { attrs: vec![] }
}
/// Add an attribute to the end of the attribute list.
pub fn push(mut self, attr: Attr) -> Self {
self.attrs.push(attr);
self
}
/// Search for an attribute of the given `name`.
///
/// _You should use this method only when a linear search makes sense._
///
/// This performs an `O(n)` linear search in the worst case.
/// Future implementations may perform an `O(1)` lookup under certain
/// circumstances,
/// but this should not be expected.
pub fn find(&self, name: QName) -> Option<&Attr> {
self.attrs.iter().find(|attr| attr.name() == name)
}
/// Returns [`true`] if the list contains no attributes.
pub fn is_empty(&self) -> bool {
self.attrs.is_empty()
}
}
impl From<Vec<Attr>> for AttrList {
fn from(attrs: Vec<Attr>) -> Self {
AttrList { attrs }
}
}
tamer: xir::tree::attr_parser_from: Integrate AttrParser This begins to integrate the isolated AttrParser. The next step will be integrating it into the larger XIRT parser. There's been considerable delay in getting this committed, because I went through quite the struggle with myself trying to determine what balance I want to strike between Rust's type system; convenience with parser combinators; iterators; and various other abstractions. I ended up being confounded by trying to maintain the current XmloReader abstraction, which is fundamentally incompatible with the way the new parsing system works (streaming iterators that do not collect or perform heap allocations). There'll be more information on this to come, but there are certain things that will be changing. There are a couple problems highlighted by this commit (not in code, but conceptually): 1. Introducing Option here for the TokenParserState doesn't feel right, in the sense that the abstraction is inappropriate. We should perhaps introduce a new variant Parsed::Done or something to indicate intent, rather than leaving the reader to have to read about what None actually means. 2. This turns Parsed into more of a statement influencing control flow/logic, and so should be encapsulated, with an external equivalent of Parsed that omits variants that ought to remain encapsulated. 3. TokenStreamState is true, but these really are the actual parsers; TokenStreamParser is more of a coordinator, and helps to abstract away some of the common logic so lower-level parsers do not have to worry about it. But calling it TokenStreamState is both a bit confusing and is an understatement---it _does_ hold the state, but it also holds the current parsing stack in its variants. Another thing that is not yet entirely clear is whether this AttrParser ought to care about detection of duplicate attributes, or if that should be done in a separate parser, perhaps even at the XIR level. The same can be said for checking for balanced tags. By pushing it to TokenStream in XIR, we would get a guaranteed check regardless of what parsers are used, which is attractive because it reduces the (almost certain-to-otherwise-occur) risk that individual parsers will not sufficiently check for semantically valid XML. But it does _potentially_ match error recovery more complicated. But at the same time, perhaps more specific parsers ought not care about recovery at that level. Anyway, point being, more to come, but I am disappointed how much time I'm spending considering parsing, given that there are so many things I need to move onto. I just want this done right and in a way that feels like it's working well with Rust while it's all in working memory, otherwise it's going to be a significant effort to get back into. DEV-11268
2021-12-10 14:13:02 -05:00
impl FromIterator<Attr> for AttrList {
fn from_iter<T: IntoIterator<Item = Attr>>(iter: T) -> Self {
iter.into_iter().collect::<Vec<Attr>>().into()
}
}
impl<const N: usize> From<[Attr; N]> for AttrList {
fn from(attrs: [Attr; N]) -> Self {
AttrList {
attrs: attrs.into(),
}
}
}
#[cfg(test)]
mod test {
use crate::span::DUMMY_CONTEXT as DC;
use super::*;
// See docblock for [`AttrSpan`].
const A: Span = DC.span(5, 13); // Entire attribute token
const K: Span = DC.span(5, 3); // Key
const V: Span = DC.span(14, 3); // Value without quotes
const Q: Span = DC.span(13, 5); // Value with quotes
#[test]
fn attr_span_token() {
assert_eq!(AttrSpan(K, V).span(), A);
}
#[test]
fn attr_span_value_with_quotes() {
assert_eq!(AttrSpan(K, V).value_span_with_quotes(), Q);
}
#[test]
fn attr_span_key() {
assert_eq!(AttrSpan(K, V).key_span(), K);
}
#[test]
fn attr_span_value() {
assert_eq!(AttrSpan(K, V).value_span(), V);
}
}