Commit Graph

32 Commits (8d618654d4074b26b21db959f84f39ec4b1ad086)

Author SHA1 Message Date
Mike Gerwitz 954b5a2795 Copyright year and name update
Ryan Specialty Group (RSG) rebranded to Ryan Specialty after its IPO.
2023-01-20 23:37:30 -05:00
Mike Gerwitz e6640c0019 tamer: Integrate clippy
This invokes clippy as part of `make check` now, which I had previously
avoided doing (I'll elaborate on that below).

This commit represents the changes needed to resolve all the warnings
presented by clippy.  Many changes have been made where I find the lints to
be useful and agreeable, but there are a number of lints, rationalized in
`src/lib.rs`, where I found the lints to be disagreeable.  I have provided
rationale, primarily for those wondering why I desire to deviate from the
default lints, though it does feel backward to rationalize why certain lints
ought to be applied (the reverse should be true).

With that said, this did catch some legitimage issues, and it was also
helpful in getting some older code up-to-date with new language additions
that perhaps I used in new code but hadn't gone back and updated old code
for.  My goal was to get clippy working without errors so that, in the
future, when others get into TAMER and are still getting used to Rust,
clippy is able to help guide them in the right direction.

One of the reasons I went without clippy for so long (though I admittedly
forgot I wasn't using it for a period of time) was because there were a
number of suggestions that I found disagreeable, and I didn't take the time
to go through them and determine what I wanted to follow.  Furthermore, it
was hard to make that judgment when I was new to the language and lacked
the necessary experience to do so.

One thing I would like to comment further on is the use of `format!` with
`expect`, which is also what the diagnostic system convenience methods
do (which clippy does not cover).  Because of all the work I've done trying
to understand Rust and looking at disassemblies and seeing what it
optimizes, I falsely assumed that Rust would convert such things into
conditionals in my otherwise-pure code...but apparently that's not the case,
when `format!` is involved.

I noticed that, after making the suggested fix with `get_ident`, Rust
proceeded to then inline it into each call site and then apply further
optimizations.  It was also previously invoking the thread lock (for the
interner) unconditionally and invoking the `Display` implementation.  That
is not at all what I intended for, despite knowing the eager semantics of
function calls in Rust.

Anyway, possibly more to come on that, I'm just tired of typing and need to
move on.  I'll be returning to investigate further diagnostic messages soon.
2023-01-20 23:37:29 -05:00
Mike Gerwitz 56d1ecf0a3 tamer: Air{Token=>}
Consistency with `Nir` et al.

DEV-13430
2022-12-13 14:36:38 -05:00
Mike Gerwitz daeaade53c tamer: tamec: Expose ASG context in lowering pipeline
The previous commit had the ASG implicitly constructed and then
discarded.  This will keep it around, which will be necessary not only for
imports, but for passing the ASG off to the next phases of lowering.

DEV-13429
2022-12-13 13:46:31 -05:00
Mike Gerwitz aa1ca06a0e tamer: tamec: Introduce NIR->AIR->ASG lowering
This does not yet yield the produces ASG, but does set up the lowering
pipeline to prepare to produce it.  It's also currently a no-op, with
`NirToAsg` just yielding `Incomplete`.

The goal is to begin to move toward vertical slices for TAMER as I start to
return to the previous approach of a handoff with the old compiler.  Now
that I've gained clarity from my previous failed approach (which I
documented in previous commits), I feel that this is the best way forward
that will allow me to incrementally introduce more fine-grained performance
improvements, at the cost of some throwaway work as this progresses.  But
the cost of delay with these build times is far greater.

DEV-13429
2022-12-13 13:37:07 -05:00
Mike Gerwitz 8d2d273932 tamer: nir::interp: Integrate NIR interpolation into lowering pipeline
This is the culmination of all the recent work---the third attempt at trying
to integrate this.  It ended up much cleaner than what was originally going
to be done, but only after gutting portions of the system and changing my
approach to how NIR is parsed (WRT attributes).  See prior commits for more
information.

The final step is to fill the error branches with actual errors rather than
`todo!`s.

What a relief.

DEV-13156
2022-12-05 16:32:00 -05:00
Mike Gerwitz 6d39474127 tamer: NIR re-simplification
Alright, this has been a rather tortured experience.  The previous commit
began to state what is going on.

This is reversing a lot of prior work, with the benefit of
hindsight.  Little bit of history, for the people who will probably never
read this, but who knows:

As noted at the top of NIR, I've long wanted a very simple set of general
primitives where all desugaring is done by the template system---TAME is a
metalanguage after all.  Therefore, I never intended on having any explicit
desugaring operations.

But I didn't have time to augment the template system to support parsing on
attribute strings (nor am I sure if I want to do such a thing), so it became
clear that interpolation would be a pass in the compiler.  Which led me to
the idea of a desugaring pass.

That in turn spiraled into representing the status of whether NIR was
desugared, and separating primitives, etc, which lead to a lot of additional
complexity.  The idea was to have a Sugared and Plan NIR, and further within
them have symbols that have latent types---if they require interpolation,
then those types would be deferred until after template expansion.

The obvious problem there is that now:

  1. NIR has the complexity of various types; and
  2. Types were tightly coupled with NIR and how it was defined in terms of
     XML destructuring.

The first attempt at this didn't go well: it was clear that the symbol types
would make mapping from Sugared to Plain NIR very complicated.  Further,
since NIR had any number of symbols per Sugared NIR token, interpolation was
a pain in the ass.

So that lead to the idea of interpolating at the _attribute_ level.  That
seemed to be going well at first, until I realized that the token stream of
the attribute parser does not match that of the element parser, and so that
general solution fell apart.  It wouldn't have been great anyway, since then
interpolation was _also_ coupled to the destructuring of the document.

Another goal of mine has been to decouple TAME from XML.  Not because I want
to move away from XML (if I did, I'd want S-expressions, not YAML, but I
don't think the team would go for that).  This decoupling would allow the
use of a subset of the syntax of TAME in other places, like CSVMs and YAML
test cases, for example, if appropriate.

This approach makes sense: the grammar of TAME isn't XML, it's _embedded
within_ XML.  The XML layer has to be stripped to expose it.

And so that's what NIR is now evolving into---the stripped, bare
repsentation of TAME's language.  That also has other benefits too down the
line, like a REPL where you can use any number of syntaxes.  I intend for
NIR to be stack-based, which I'd find to be intuitive for manipulating and
querying packages, but it could have any number of grammars, including
Prolog-like for expressing Horn clauses and querying with a
Prolog/Datalog-like syntax.  But that's for the future...

The next issue is that of attribute types.  If we have a better language for
NIR, then the types can be associated with the NIR tokens, rather than
having to associate each symbol with raw type data, which doesn't make a
whole lot of sense.  That also allows for AIR to better infer types and
determine what they ought to be, and further makes checking types after
template application natural, since it's not part of NIR at all.  It also
means the template system can naturally apply to any sources.

Now, if we take that final step further, and make attributes streaming
instead of aggregating, we're back to a streaming pipeline where all
aggregation takes place on the ASG (which also resolves the memcpy concerns
worked around previously, also further simplifying `ele_parse` again, though
it sucks that I wasted that time).  And, without the symbol types getting
in the way, since now NIR has types more fundamentally associated with
tokens, we're able to interpolate on a token stream using simple SPairs,
like I always hoped (and reverted back to in the previous commit).

Oh, and what about that desugaring pass?  There's the issue of how to
represent such a thing in the type system---ideally we'd know statically
that desugaring always lowers into a more primitive NIR that reduces the
mapping that needs to be done to AIR.  But that adds complexity, as
mentioned above.  The alternative is to just use the templat system, as I
originally wanted to, and resolve shortcomings by augmenting the template
system to be able to handle it.  That not only keeps NIR and the compiler
much simpler, but exposes more powerful tools to developers via TAME's
metalanguage, if such a thing is appropriate.

Anyway, this creates a system that's far more intuitive, and far
simpler.  It does kick the can to AIR, but that's okay, since it's also
better positioned to deal with it.

Everything I wrote above is a thought dump and has not been proof-read, so
good luck!  And lets hope this finally works out...it's actually feeling
good this time.  The journey was necessary to discover and justify what came
out of it---everything I'm stripping away was like a cocoon, and within it
is a more beautiful and more elegant TAME.

DEV-13346
2022-12-01 11:09:25 -05:00
Mike Gerwitz d195eedacb tamer: nir: Sugared and plain flavors
This introduces the concept of sugared NIR and provides the boilerplate for
a desugaring pass.  The earlier commits dealing with cleaning up the
lowering pipeline were to support this work, in particular to ensure that
reporting and recovery properly applied to this lowering operation without
adding a ton more boilerplate.

DEV-13158
2022-10-26 14:19:19 -04:00
Mike Gerwitz dbe834b48a tamer: tamec: Remove lowering pipeline refactoring comment
I'm struggling to go much further yet without sorting out some other things
first with regards to mutable `Context` and, in particular, the ASG.

I'm going to pause on refactoring the lowering pipeline---it's been improved
significantly with the recent work---and I will continue in the next few
weeks.

DEV-13158
2022-10-26 12:44:20 -04:00
Mike Gerwitz 7c4c0ebdda tamer: parse::lower: Separate error types for lowering and return
Lowering errors in tamec end up utilizing recovery and reporting, so there
is a distinction between recoverable and unrecoverable errors.

tameld aborts on the first error, since recovery is not currently
supported (we'll want to add it, since tameld should output e.g. lists of
unresolved externs).

Note that tamec does not yet handle `FinalizeError` like tameld because it
uses `Lower::lower`, which does not yet finalize (though it does in practice
when it reaches the end of the stream and auto-finalizes, but that is
widened into a `ParseError`).

DEV-13158
2022-10-26 12:44:20 -04:00
Mike Gerwitz 7e62276907 tamer: Revert "tamer: diagnose::report::Report: {Mutable=>immutable} self reference"
This reverts commit 85ec626fcd804eb2fac3fd6f0339182554f72cfd.

This revert had to be modified to work alongside other changes.  Interior
mutability is fortunately no longer needed after the previous commit which
allows reporting to occur in a single place in the lowering pipeline (at the
terminal parser).

DEV-13158
2022-10-26 12:44:18 -04:00
Mike Gerwitz 1c181fe546 tamer: parse::lower: Propagate widened errors to terminal parser
The term "terminal parser" isn't formalized yet in the system, but is meant
to refer to the innermost parser that is responsible for pulling tokens
through the lowering pipeline.

This approach is more of what one would expect when dealing with
`Result`-like monads---we are effectively chaining the inner operation while
propagating errors to short-circuit lowering and let the caller decide
whether recovery ought to be permitted with diagnostic messages.  This will
become more clear as it is further refactored.

This also means that the previous changes for introducing interior
mutability for a shared mutable `Reporter` can be reverted, which is great,
since that approach was antithetical to how the streaming pipeline
operates (and introduces awkward mutable state into an
otherwise-mostly-immutable system).

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 2ccdaf80fe tamer: diagnose::report: Error tracking
This extracts error tracking into the Reporter itself, which is already
shared between lowering operations.  This can then be used to display the
number of errors.

A new formatter (in tamer::fmt) will be added to handle the singular/plural
conversion in place of "error(s)" in the future; I have more important
things to work on right now.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz f049da4496 tamer: tamec: Apply reporting (and continuing) to XirToXirf failure
Previously these errors would immediately abort.

This results in some duplicate code, but it's beginning to derive a common
implementation.  Check out the commits that follow; this is really an
intermediate refactoring state.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 733f44a616 tamer: diagnose::report::Report: {Mutable=>immutable} self reference
VisualReporter now uses interior mutability so that we can hold multiple
references to it for upcoming lowering pipeline changes.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz a6e72b87f7 tamer: tamec: Extract compilation from main
Another baby step.  The small commits are intended to allow comprehension of
what changes when looking at the diffs.

This also removes a comment stating that errors do not fail compilation,
since they most certainly do.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 20ea83af1a tamer: tamec: Extract source reading and writing
This begins refactoring the lowering pipeline to begin to obviate
abstraction boundaries.  The lowering pipeline is the backbone of the
system, and so it needs to become clear and self-documenting, which will
take a little bit of work.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 3456bd593a tamer: tamec: Fail with non-zero status if any NIR parsing errors
This is a quick-and-dirty change.  The lowering pipeline needs a proper
abstraction, but I'm about to be on vacation at the end of the week and
would like to get NIR->AIR lowering started before I consider that
abstraction further, so this will do for now.

NIR parsing has been tested in production without failing for over a week.

DEV-7145
2022-09-19 10:11:47 -04:00
Mike Gerwitz 419b24f251 tamer: Introduce NIR (accepting only)
This introduces NIR, but only as an accepting grammar; it doesn't yet emit
the NIR IR, beyond TODOs.

This modifies `tamec` to, while copying XIR, also attempt to lower NIR to
produce parser errors, if any.  It does not yet fail compilation, as I just
want to be cautious and observe that everything's working properly for a
little while as people use it, before I potentially break builds.

This is the culmination of months of supporting effort.  The NIR grammar is
derived from our existing TAME sources internally, which I use for now as a
test case until I introduce test cases directly into TAMER later on (I'd do
it now, if I hadn't spent so much time on this; I'll start introducing tests
as I begin emitting NIR tokens).  This is capable of fully parsing our
largest system with >900 packages, as well as `core`.

`tamec`'s lowering is a mess; that'll be cleaned up in future commits.  The
same can be said about `tameld`.

NIR's grammar has some initial documentation, but this will improve over
time as well.

The generated docs still need some improvement, too, especially with
generated identifiers; I just want to get this out here for testing.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 8d92667388 tamer: Integrate xir::reader as a parser in the lowering pipeline
This allows `XmlXirReader` to be used in a `Lower` operation, just as
everything else, bringing me one step closer to a pipeline that can be
concisely represented; this is finally beginning to unify in a clear way,
though it is still a bit of a mess.

This causes `XmlXirReader` to _act_ like a `parse::Parser` in that it yields
a `ParsedResult`, but it does not use `parse::Parser` itself; that was the
_original_ plan: convert it into a `ParseState` where `XmlXirReader` became
a context, and force `Parser` to yield by feeding it a stream of tokens with
`repeat`, but that ended up performing poorly relative to this change.  I
did some investigation, which I might write about in the future, but for
now, this solution works just fine.

DEV-7145
2022-06-02 10:30:44 -04:00
Mike Gerwitz f218c452b9 tamer: iter::trip: Flatten Result
The `*_iter_while_ok` functions now compose like monads, flattening `Result`
at each step and drastically simplifying handling of error types.  This also
removes the bunch of `?`s at the end of the expression, and allows me to use
`?` within the callback itself.

I had originally not used `Result` as the return type of the callback
because I was not entirely sure how I was going to use them, but it's now
clear that I _always_ use `Result` as the return type, and so there's no use
in trying to be too accommodating; it can always change in the future.

This is desirable not just for cleanup, but because trying to refactor
`asg_builder` into a pair of `Parser`s is really messy to chain without
flattening, especially given some state that has to leak temporarily to the
caller.  More on that in a future commit.

DEV-11864
2022-05-20 16:08:16 -04:00
Mike Gerwitz 0281dfdf0d tamer: Remove wip-frontends feature flag
We want the new system to be used so that we can start catching any problems
that may arise.  Further changes will be flagged as necessary.

DEV-10936
2022-05-04 09:37:10 -04:00
Mike Gerwitz 1ad2fb1dc8 Copyright year update 2022
RSG (Ryan Specialty Group) recently announced a rename to Ryan Specialty (no
"Group"), but I'm not sure if the legal name has been changed yet or not, so
I'll wait on that.
2022-05-03 14:14:29 -04:00
Mike Gerwitz 3dbab881da tamer: diagnose::report: Produce Report object
Rather than writing to the provided `Write` object, this produces a `Report`
object.  While a lifetime still exists for the diagnostic data (labels,
specifically), I was able to remove the other lifetime resulting from
`ResolvedSpan` by transferring ownership of the data to the `Report`
itself.  Once actual source lines are integrated shortly, `Report` will
include those as well.

This has been a tedious process, but it's coming together.  Hopefully these
commits documenting the progressive and ugly refactoring are found useful by
some reader in the future.

DEV-12151
2022-04-27 15:00:30 -04:00
Mike Gerwitz a22e8e79f7 tamer: diagnose: Integrate resolver for source lines
This does not yet resolve columns, and omits the length of the span, but
it's starting to come together.

This is particularly exciting for me to see because I've been wanting line
numbers in TAME error messages for over a decade.

DEV-10935
2022-04-21 12:34:17 -04:00
Mike Gerwitz 725dc3fb54 tamer: tamec: Use diagnostic system for errors
This is a POC, minimal-effort integration that also creates the TamecError
sum type analogous to TameldError.

I'll work on reducing the boilerplate in the future.

A note regarding the type and boilerplate vs. dynamic dispatch, for any
future readers: the purpose of this is to be explicit about the error types
so that the system is self-documenting and it forces and understanding of
its error conditions.  `Box<dyn Error>` is basically "eh idk anything can
happen!", which is not what I'm interested in having.

DEV-10935
2022-04-20 09:42:11 -04:00
Mike Gerwitz a1a4ad3e8e tamer: Introduce context into XirReader
tamec and tameld will now both introduce a `Context` to XIR, which will use
it to create spans.

Here's an example of an error, now that it's all working well together:

  $ target/release/tameld --emit xmle -o /dev/null path/to/package.xmlo
  error: invalid preproc:sym/@dim `9` at [/../path/to/package.xmlo offset 1175451-1175452]

A future task will make this human-readable by producing line and column
numbers, and perhaps even a snippet (if not now, then eventually).

It's exciting to see this coming together finally.

DEV-10934
2022-04-08 16:16:23 -04:00
Mike Gerwitz 99aacaf7ca tamer: tamec: Replace copy with XIR parsing/writing
When wip-frontends is on, this will parse the input file using XIR and then
immediately output it again.  This makes the necessary changes to be able to
read every source file we have in our largest project, such that the output
is identical after having been formatted with `xmllint --format -` (there
are differences because e.g. whitespace between attributes is not yet
maintained).

This is performant too, with times remaining essentially identical despite
the additional work.

DEV-10413
2022-04-07 12:13:49 -04:00
Mike Gerwitz ca6ef3ed36 tamer: frontend: Begin basic XML parsing
The first step in the process is to emit the raw XML events that can then be
immediately output again to echo the results into another file.  This will
then allow us to begin parsing the input incrementally, and begin to morph
the output into a real `xmlo` file.
2021-07-27 00:37:13 -04:00
Mike Gerwitz fb8422d670 tamer: Initial frontend concept
This introduces the beginnings of frontends for TAMER, gated behind a
`wip-features` flag.

This will be introduced in stages:

  1. Replace the existing copy with a parser-based copy (echo back out the
     tokens), when the flag is on.
  2. Begin to parse portions of the source, augmenting the output xmlo (xmli
     at the moment).  The XSLT-based compiler will be modified to skip
     compilation steps as necessary.

As portions of the compilation are implemented in TAMER, they'll be placed
behind their own feature flags and stabalized, which will incrementally
remove the compilation steps from the XSLT-based system.  The result should
be substantial incremental performance improvements.

Short-term, the priorities are for loading identifiers into an IR
are (though the order may change):

  1. Echo
  2. Imports
  3. Extern declarations.
  4. Simple identifiers (e.g. param, const, template, etc).
  5. Classifications.
  6. Documentation expressions.
  7. Calculation expressions.
  8. Template applications.
  9. Template definitions.
  10. Inline templates.

After each of those are done, the resulting xmlo (xmli) will have fully
reconstructed the source document from the IR produced during parsing.
2021-07-23 22:24:08 -04:00
Mike Gerwitz 2e50af1220 Copyright year update 2021 2021-07-22 15:00:15 -04:00
Joseph Frazer 2c587e2d9d [DEV-7147] Add "tamec" executable
Add a stub executable that will eventually become a full-featured TAME
compiler. The first implementation will only copy the source file to an
intermediary file that will be compiled by the XSLT compiler.
2020-04-09 09:46:46 -04:00