Commit Graph

31 Commits (ebdae9ac38830338f862d609406c0351f1095154)

Author SHA1 Message Date
Mike Gerwitz e3a68aaf9e tamer: span: Introduce rslice, slice_{head,tail}
These are used by an upcoming commit, where I'll have much more to say about
the topic of span slicing.

DEV-13162
2023-05-05 10:25:35 -04:00
Mike Gerwitz 9eb1d226b2 tamer: span: Resolve unusued import warning for debug diagnostic 2023-03-10 14:27:58 -05:00
Mike Gerwitz e6f736298b tamer: asg::graph::visit::tree_reconstruction: New graph traversal
This begins to introduce a graph traversal useful for a source
reconstruction from the current state of the ASG.  The idea is to, after
having parsed and ingested the source through the lowering pipeline, to
re-output it to (a) prove that we have parsed correctly and (b) allow
progressively moving things from the XSLT-based compiler into TAMER.

There's quite a bit of documentation here; see that for more
information.  Generalizing this in an appropriate way took some time, but I
think this makes sense (that work began with the introduction of cross edges
in terms of the tree described by the graph's ontology).  But I do need to
come up with an illustration to include in the documentation.

DEV-13708
2023-03-10 14:27:57 -05:00
Mike Gerwitz 954b5a2795 Copyright year and name update
Ryan Specialty Group (RSG) rebranded to Ryan Specialty after its IPO.
2023-01-20 23:37:30 -05:00
Mike Gerwitz 4e3a81d7f5 tamer: asg: Bind transparent ident
This provides the initial implementation allowing an identifier to be
defined (bound to an object and made transparent).

I'm not yet entirely sure whether I'll stick with the "transparent" and
"opaque" terminology when there's also "declare" and "define", but a
`Missing` state is a type of declaration and so the distinction does still
seem to be important.

There is still work to be done on `ObjectIndex::<Ident>::bind_definition`,
which will follow.  I'm going to be balancing work to provide type-level
guarantees, since I don't have the time to go as far as I'd like.

DEV-13597
2023-01-20 23:37:29 -05:00
Mike Gerwitz e6640c0019 tamer: Integrate clippy
This invokes clippy as part of `make check` now, which I had previously
avoided doing (I'll elaborate on that below).

This commit represents the changes needed to resolve all the warnings
presented by clippy.  Many changes have been made where I find the lints to
be useful and agreeable, but there are a number of lints, rationalized in
`src/lib.rs`, where I found the lints to be disagreeable.  I have provided
rationale, primarily for those wondering why I desire to deviate from the
default lints, though it does feel backward to rationalize why certain lints
ought to be applied (the reverse should be true).

With that said, this did catch some legitimage issues, and it was also
helpful in getting some older code up-to-date with new language additions
that perhaps I used in new code but hadn't gone back and updated old code
for.  My goal was to get clippy working without errors so that, in the
future, when others get into TAMER and are still getting used to Rust,
clippy is able to help guide them in the right direction.

One of the reasons I went without clippy for so long (though I admittedly
forgot I wasn't using it for a period of time) was because there were a
number of suggestions that I found disagreeable, and I didn't take the time
to go through them and determine what I wanted to follow.  Furthermore, it
was hard to make that judgment when I was new to the language and lacked
the necessary experience to do so.

One thing I would like to comment further on is the use of `format!` with
`expect`, which is also what the diagnostic system convenience methods
do (which clippy does not cover).  Because of all the work I've done trying
to understand Rust and looking at disassemblies and seeing what it
optimizes, I falsely assumed that Rust would convert such things into
conditionals in my otherwise-pure code...but apparently that's not the case,
when `format!` is involved.

I noticed that, after making the suggested fix with `get_ident`, Rust
proceeded to then inline it into each call site and then apply further
optimizations.  It was also previously invoking the thread lock (for the
interner) unconditionally and invoking the `Display` implementation.  That
is not at all what I intended for, despite knowing the eager semantics of
function calls in Rust.

Anyway, possibly more to come on that, I'm just tired of typing and need to
move on.  I'll be returning to investigate further diagnostic messages soon.
2023-01-20 23:37:29 -05:00
Mike Gerwitz 646633883f tamer: Initial concept for AIR/ASG Expr
This begins to place expressions on the graph---something that I've been
thinking about for a couple of years now, so it's interesting to finally be
doing it.

This is going to evolve; I want to get some things committed so that it's
clear how I'm moving forward.  The ASG makes things a bit awkward for a
number of reasons:

  1. I'm dealing with older code where I had a different model of doing
       things;
  2. It's mutable, rather than the mostly-functional lowering pipeline;
  3. We're dealing with an aggregate ever-evolving blob of data (the graph)
       rather than a stream of tokens; and
  4. We don't have as many type guarantees.

I've shown with the lowering pipeline that I'm able to take a mutable
reference and convert it into something that's both functional and
performant, where I remove it from its container (an `Option`), create a new
version of it, and place it back.  Rust is able to optimize away the memcpys
and such and just directly manipulate the underlying value, which is often a
register with all of the inlining.

_But_ this is a different scenario now.  The lowering pipeline has a narrow
context.  The graph has to keep hitting memory.  So we'll see how this
goes.  But it's most important to get this working and measure how it
performs; I'm not trying to prematurely optimize.  My attempts right now are
for the way that I wish to develop.

Speaking to #4 above, it also sucks that I'm not able to type the
relationships between nodes on the graph.  Rather, it's not that I _can't_,
but a project to created a typed graph library is beyond the scope of this
work and would take far too much time.  I'll leave that to a personal,
non-work project.  Instead, I'm going to have to narrow the type any time
the graph is accessed.  And while that sucks, I'm going to do my best to
encapsulate those details to make it as seamless as possible API-wise.  The
performance hit of performing the narrowing I'm hoping will be very small
relative to all the business logic going on (a single cache miss is bound to
be far more expensive than many narrowings which are just integer
comparisons and branching)...but we'll see.  Introducing branching sucks,
but branch prediction is pretty damn good in modern CPUs.

DEV-13160
2022-12-22 14:33:28 -05:00
Mike Gerwitz 97685fd146 tamer: span::Span::merge: New method
This will be used for expression start and end spans to merge into a span
that represents the entirety of the expression; see future commits for its
use.

Though, this has been generalized further than that to ensure that it makes
sense in any use case, to avoid potential pitfalls.

DEV-13160
2022-12-21 12:35:01 -05:00
Mike Gerwitz 5c5041f90e tamer: nir::desugar::interp: Proper span offsets
The spans were previously not being calculated relative to the offset of the
original symbol span.  Tests were passing because all of those spans began
at offset 0.

DEV-13156
2022-11-08 00:55:45 -05:00
Mike Gerwitz 8f3301431c tamer: span::dummy: New module to hold DUMMY_SPAN and derivatives
Various DUMMY_SPAN-derived spans are used by many test cases, so this
finally extracts them---something I've been meaning to do for some time.

This also places DUMMY_SPAN behind a `cfg(test)` directive to ensure that it
is _only_ used in tests; UNKNOWN_SPAN should be used when a span is actually
unknown, which may also be the case during development.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 495c1438fd tamer: Consistent span diagram representation
I'll document it more formally eventually, but this settles on a mix of the
two: square brackets and dashes for intervals, `+` for intersecting lines,
byte offsets below interval endpoints, and names below that.

The docblock for `Span` itself iss still off; I'll probably just take one of
the test cases and paste it there at some point.

DEV-7145
2022-06-06 11:32:35 -04:00
Mike Gerwitz bba181f573 tamer: xir::attr::Attr: Introduce AttrSpan
This replaces a tuple with a tuple struct that allows for calculating more
complete span information, such as the span encompassing the entire
attribute and the value span including the surrounding quotes.

This includes logic that ought to be abstracted into `Span` itself, and it's
not as formal as I'd like it to be (e.g. not ensuring context), but this is
a good starting point.

Note that parsers call `Token::span`, which in turn calculates the attribute
span, each time an attribute is encountered during lowering.  But Rust does
a good job at optimizing away unnecessary operations, so this didn't have an
observable impact on time.

DEV-7145
2022-06-06 11:31:28 -04:00
Mike Gerwitz 1ad2fb1dc8 Copyright year update 2022
RSG (Ryan Specialty Group) recently announced a rename to Ryan Specialty (no
"Group"), but I'm not sure if the legal name has been changed yet or not, so
I'll wait on that.
2022-05-03 14:14:29 -04:00
Mike Gerwitz d05bcaab03 tamer: {Resolved,Span}::{ctx=>context}: Rename
This is just to provide clarity.  `ctx` is not so widely used that we
benefit from such a short identifier, and it's not worth the cognitive
burden of people unfamiliar with what it may mean.

DEV-12151
2022-04-26 10:52:32 -04:00
Mike Gerwitz e555955450 tamer: span::Span::endpoints_saturated: New method
This gets rid of the `Option` and is used in the diagnostic system (next
commit).

DEV-10935
2022-04-21 14:15:25 -04:00
Mike Gerwitz 1b02e77537 tamer: span (SpanOffsetSize, SpanLenSize): New type aliases
Callers can use these types instead of having to reference globals.

DEV-10935
2022-04-20 09:42:13 -04:00
Mike Gerwitz a77eb7d937 tamer: span: Minor test refactoring
Just some cleanup based on some new conventions, now that I'm about to make
some changes.

DEV-10935
2022-04-20 09:42:12 -04:00
Mike Gerwitz eaa8133d21 tamer: diagnose: Introduction of diagnostic system
This is a working concept that will continue to evolve.  I wanted to start
with some basic output before getting too carried away, since there's a lot
of potential here.

This is heavily influenced by Rust's helpful diagnostic messages, but will
take some time to realize a lot of the things that Rust does.  The next step
will be to resolve line and column numbers, and then possibly include
snippets and underline spans, placing the labels alongside them.  I need to
balance this work with everything else I have going on.

This is a large commit, but it converts the existing Error Display impls
into Diagnostic.  This separation is a bit verbose, so I'll see how this
ends up evolving.

Diagnostics are tied to Error at the moment, but I imagine in the future
that any object would be able to describe itself, error or not, which would
be useful in the future both for the Summary Page and for query
functionality, to help developers understand the systems they are writing
using TAME.

Output is integrated into tameld only in this commit; I'll add tamec
next.  Examples of what this outputs are available in the test cases in this
commit.

DEV-10935
2022-04-13 15:22:46 -04:00
Mike Gerwitz 702b5ebb23 tamer: span: Remove PathIndex
We can just use PathSymbolId directly and simplify things.  Typing can (and
should) happen on the symbol itself, and if we want a separate symbol type,
it ought to have its own interner.

For now, it doesn't, and having this extra type is just a PITA.

DEV-10935
2022-04-13 09:59:11 -04:00
Mike Gerwitz ab181670b5 tamer: xir::reader: Initial introduction of spans
This is a large change, and was a bit of a tedious one, given the
comprehensive tests.

This introduces proper offsets and lengths for spans, with the exception of
some quick-xml errors that still need proper mapping.  Further, this still
uses `UNKNOWN_CONTEXT`, which will be resolved shortly.

This also introduces `SpanlessError`, which `Error` explicitly _does not_
implement `From<SpanlessError>` for---this forces the caller to provide a
span before the error is compatable with the return value, ensuring that
spans will actually be available rather than forgotten for errors.  This is
important, given that errors are generally less tested than the happy path,
and errors are when users need us the most (so, need span information).

Further, I had to use pointer arithmetic in order to calculate many of the
spans, because quick-xml does not provide enough information.  There's no
safety considerations here, and the comprehensive unit test will ensure
correct behavior if the implementation changes in the future.

I would like to introduce typed spans at some point---I made some
opinionated choices when it comes to what the spans ought to
represent.  Specifically, whether to include the `<` or `>` with the open
span (depends), whether to include quotes with attribute values (no),
and some other details highlighted in the test cases.  If we provide typed
spans, then we could, knowing the type of span, calculate other spans on
request, e.g. to include or omit quotes for attributes.  Different such
spans may be useful in different situations when presenting information to
the user.

This also highlights gaps in the tokens emitted by XIR, such as whitespace
between attributes, the `=` between name and value, and so on.  These are
important when it comes to code formatting, so that we can reliably
reconstruct the XML tree, but it's not important right now.  I anticipate
future changes would allow the XIR reader to be configured (perhaps via
generics, like a strategy-type pattern) to optionally omit these tokens if
desired.

Anyway, more to come.

DEV-10934
2022-04-08 13:59:37 -04:00
Mike Gerwitz f42288f3a2 tamer: obj::xmlo::reader: Begin symbol table parsing
This wasn't the simplest thing to start with, but I wanted to explore
something with a higher level of complexity.  There is some boilerplate to
observe here, including:

  1. The state stitching (as I guess I'm calling it now) of SymtableState
     with XmloReaderState is all boilerplate and requires no lookahead,
     presenting an abstraction opportunity that I was holding off on
     previously (attr parsing for XIRF requires lookahead).
  2. This is simply collecting attributes into a struct.  This can be
     abstracted away in the future.
  3. Creating stub parsers to verify that generics are stitched rather than
     being tightly coupled with another state is boilerplate that maybe can
     be abstracted away after a pattern is observed in future tests.

DEV-10863
2022-03-29 11:14:47 -04:00
Mike Gerwitz aba89f809d tamer: xir::parse: UnexpectedEof Span at final offset
I'm not rendering errors yet in practice, so this wouldn't have been
noticed, but we want error messages to reference the final byte in a file on
EOF, not the offset of the last-encountered token, which would be confusing.

This doesn't _directly_ pertain to what I'm working on; I just happened to
notice it.

DEV-10863
2022-03-17 21:33:05 -04:00
Mike Gerwitz ce48a654b1 tamer: span::Span::offset_add: Make const
This behavior is unchanged, but it allows us to create more constant spans
for testing.  For example:

  const S = DUMMY_SPAN.offset_add(1).unwrap();

This, in turn, will allow for removing lazy_static! for tests that use it
for span generation.

DEV-10863
2022-03-16 14:16:28 -04:00
Mike Gerwitz 7873d46afb tamer: Replace all &'static str in errors with SymbolId
Now that SymbolId implements Display and resolves, this works out well.
2021-10-11 15:39:53 -04:00
Mike Gerwitz 7e9271e189 tamer: span: Primitive Display impl
This outputs enough information to be a little bit useful in the event of an
error.  In the future, we'll want to provide a (likely non-Display)
implementation that provides line number and source file context with
the problem characters indicated, like Rust.
2021-10-11 14:14:43 -04:00
Mike Gerwitz cde08b125c tamer: span (DUMMY_SPAN): New constant
Rather than having to use lazy_static! in all these tests, we can derive an
unlimited number of dummy spans from this one using e.g. `offset_add`.
2021-10-11 10:29:58 -04:00
Mike Gerwitz cf239531e0 tamer: span (offset_add): New method
More will come in the future, including the ability to add two spans.
2021-10-11 10:28:47 -04:00
Mike Gerwitz de3d7ef393 tamer: span: Introduce twospan
The intent is to support the composition and decomposition of spans such
that (A, B) is as documented here.  This only performs the trivial case for
the sake of providing a convenient API when the developer would otherwise
just type (S, S).
2021-10-11 09:56:48 -04:00
Mike Gerwitz 6864fbc1cd tamer: Start of XIR-based xmle writer
This has been a long time coming, and has been repeatedly stashed as other
parts of the system have evolved to support it.  The introduction of the XIR
tree was to write tests for this (which are sloppy atm).

This currently writes out the `xmle` header and _most_ of the `l:dep`
section; it's missing the object-type-specific attributes.  There is,
relatively speaking, not much more work to do here.

The feature flag `wip-xir-xmle-writer` was introduced to toggle this system
in place of `XmleWriter`.  Initial benchmarks show that it will be
competitive with the quick-xml-based writer, but remember that is not the
goal: the purpose of this is to test XIR in a production system before we
continue to implement it for a frontend, and to refactor so that we do not
have multiple implementations writing XML files (once we echo the source XML
files).

I'm excited to get this done with so that I can move on.  This has been
rather exhausting.
2021-09-28 14:52:53 -04:00
Mike Gerwitz e91aeef478 tamer: Remove Ix generalization throughout system
This had the writing on the wall all the same as the `'i` interner lifetime
that came before it.  It was too much of a maintenance burden trying to
accommodate both 16-bit and 32-bit symbols generically.

There is a situation where we do still want 16-bit symbols---the
`Span`.  Therefore, I have left generic support for symbol sizes, as well as
the different global interners, but `SymbolId` now defaults to 32-bit, as
does `Asg`.  Further, the size parameter has been removed from the rest of
the code, with the exception of `Span`.

This cleans things up quite a bit, and is much nicer to work with.  If we
want 16-bit symbols in the future for packing to increase CPU cache
performance, we can handle that situation then in that specific case; it's a
premature optimization that's not at all worth the effort here.
2021-09-23 14:52:54 -04:00
Mike Gerwitz 0ff0f88e5f tamer: Introduce span
This is an initial implementation optimized for expected use
cases.  Hopefully that pans out and doesn't come back to bite me.

Regarding the context: it only allows for interned paths atm, which are
strings (and so much be valid UTF-8, which is fine for us, but sucks for
something more general-purpose).  I'll be curious if the context needs
extension later on, or if different contexts will be stored in IRs (e.g. to
store a template application site as well as the location of the expansion
within the template body).
2021-08-13 15:16:39 -04:00