Commit Graph

1421 Commits (26aaf6efc159002fe028871dfb82f19b5240e075)

Author SHA1 Message Date
Mike Gerwitz 26aaf6efc1 tamer: parse::error::ParseError: Extract some variants into FinalizeError
This helps to clarify the situations under which these errors can occur, and
the generality also helps to show why the inner types are as they
are (e.g. use of `String`).

But more importantly, this allows for an error type in `finalize` that is
detached from the `ParseState`, which will be able to be utilized in the
lowering pipeline as a more general error distinguishable from other
lowering errors.  At the moment I'm maintaining BC, but a following commit
will demonstrate the use case to introduce recoverable vs. non-recoverable
errors.

DEV-13158
2022-10-26 12:44:19 -04:00
Mike Gerwitz 2087672c47 tamer: parse::parser::finalize: Introduce FinalizedParser
This newtype allows a caller to prove (using types) that a parser of a given
type (`ParseState`) has been finalized.

This will be used by the lowering pipeline to ensure that all parsers in the
pipeline end up getting finalized (as you can see from a TODO added in the
code, one of them is missing).  The lack of such a type was an oversight
during the (rather stressed) development of the parsing system, and I
shouldn't need to resort to unit tests to verify that parsers have been
finalized.

DEV-13158
2022-10-26 12:44:19 -04:00
Mike Gerwitz 7e62276907 tamer: Revert "tamer: diagnose::report::Report: {Mutable=>immutable} self reference"
This reverts commit 85ec626fcd804eb2fac3fd6f0339182554f72cfd.

This revert had to be modified to work alongside other changes.  Interior
mutability is fortunately no longer needed after the previous commit which
allows reporting to occur in a single place in the lowering pipeline (at the
terminal parser).

DEV-13158
2022-10-26 12:44:18 -04:00
Mike Gerwitz 1c181fe546 tamer: parse::lower: Propagate widened errors to terminal parser
The term "terminal parser" isn't formalized yet in the system, but is meant
to refer to the innermost parser that is responsible for pulling tokens
through the lowering pipeline.

This approach is more of what one would expect when dealing with
`Result`-like monads---we are effectively chaining the inner operation while
propagating errors to short-circuit lowering and let the caller decide
whether recovery ought to be permitted with diagnostic messages.  This will
become more clear as it is further refactored.

This also means that the previous changes for introducing interior
mutability for a shared mutable `Reporter` can be reverted, which is great,
since that approach was antithetical to how the streaming pipeline
operates (and introduces awkward mutable state into an
otherwise-mostly-immutable system).

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 2ccdaf80fe tamer: diagnose::report: Error tracking
This extracts error tracking into the Reporter itself, which is already
shared between lowering operations.  This can then be used to display the
number of errors.

A new formatter (in tamer::fmt) will be added to handle the singular/plural
conversion in place of "error(s)" in the future; I have more important
things to work on right now.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz f049da4496 tamer: tamec: Apply reporting (and continuing) to XirToXirf failure
Previously these errors would immediately abort.

This results in some duplicate code, but it's beginning to derive a common
implementation.  Check out the commits that follow; this is really an
intermediate refactoring state.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 733f44a616 tamer: diagnose::report::Report: {Mutable=>immutable} self reference
VisualReporter now uses interior mutability so that we can hold multiple
references to it for upcoming lowering pipeline changes.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz a6e72b87f7 tamer: tamec: Extract compilation from main
Another baby step.  The small commits are intended to allow comprehension of
what changes when looking at the diffs.

This also removes a comment stating that errors do not fail compilation,
since they most certainly do.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 20ea83af1a tamer: tamec: Extract source reading and writing
This begins refactoring the lowering pipeline to begin to obviate
abstraction boundaries.  The lowering pipeline is the backbone of the
system, and so it needs to become clear and self-documenting, which will
take a little bit of work.

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 8c32967cbf tamer: Cargo.toml: Sort dependencies
This always annoys me when I add a dependency and I don't know where I ought
to put it.

Anyway, I was originally going to add the `regex` crate, but with further
planning, I may not end up having use for it.  Nonetheless, at least this is
consistent.
2022-10-18 14:48:14 -04:00
Brandon Ellis b3d8f6c4cd RELEASES.md: Update for v19.1.0 2022-09-22 12:23:13 -04:00
Brandon Ellis 00f46b0032 [DEV-12990] Add gt, gte, lt, lte operators to if/unless
This includes updating Tamer's parser to account for the new
operator possibilities.
2022-09-22 11:38:06 -04:00
Mike Gerwitz 25babde084 .gitlab-ci.yml (build): Re-add tamer/target/doc
This was accidentally removed in a previous commit to reduce artifact sizes.
2022-09-20 09:52:42 -04:00
Mike Gerwitz 80d7de7376 tamer: nir: Remove token `todo!`s
Just preparing to actually define NIR itself.  The _grammar_ has been
represented (derived from our internal systems, using them as a test case),
but the IR itself has not yet received a definition.

DEV-7145
2022-09-19 16:21:42 -04:00
Mike Gerwitz 3456bd593a tamer: tamec: Fail with non-zero status if any NIR parsing errors
This is a quick-and-dirty change.  The lowering pipeline needs a proper
abstraction, but I'm about to be on vacation at the end of the week and
would like to get NIR->AIR lowering started before I consider that
abstraction further, so this will do for now.

NIR parsing has been tested in production without failing for over a week.

DEV-7145
2022-09-19 10:11:47 -04:00
Mike Gerwitz 5403dd06c6 tamer: Provide links to `tame{c,ld}`
DEV-7145
2022-09-19 10:04:40 -04:00
Mike Gerwitz 9966b82b9d tamer: nir::parse: Grammar summary docs
This is intended to provide just enough information to help elucidate how
the system works and why.

DEV-7145
2022-09-19 09:26:38 -04:00
Mike Gerwitz dcb42b6e4b tamer: xir::parse: Improvements to generated docs for NIR attributes
This hides the internal state machine and provides better language for what
remains.

DEV-7145
2022-09-16 13:37:46 -04:00
Mike Gerwitz 1dc691160b tamer: nir: Re-define "NIR"
This was originally the "noramlized" IR, but that's not possible to do
without template expansion, which is going to happen at a later point.  So,
this is just "NIR", pronounced "near", which is an IR that is "near" to the
source code.  You can define it was "Near IR" if you want, but it's just a
homonym with a not-quite-defined acronym to me.

DEV-7145
2022-09-16 09:59:38 -04:00
Mike Gerwitz f9bdcc2775 tamer: xir::parse::ele: Remove `*Error_` types
A type alias was added for BC before errors were hoisted out in a previous
commit, but they are unnecessary because of the associated type on
`ParseState`.

This also corrects the long-existing issue of using generated identifiers in
tests.

DEV-7145
2022-09-15 16:10:47 -04:00
Mike Gerwitz 071c94790f tamer: xir::ele::parse: Formatting: remove a level of indentation
This moves `paste::paste!` up a line and reduces a level of indentation,
since it's so squished.  Aside from docblock reformatting, there are no
other changes.

DEV-7145
2022-09-15 16:09:49 -04:00
Mike Gerwitz b3f4378517 tamer: xir::parse::ele: Hoist NT Display from `ele_parse!` macro
This slims out the macro even further.  It does result in an
awkwardly-placed `PhantomData` because I don't want to add another variant
that isn't actually used (since they represent states).

DEV-7145
2022-09-14 16:34:59 -04:00
Mike Gerwitz 80f29e9420 tamer: xir::parse::ele: Hoist NtState out of `ele_parse!` macro
This does the same as before with SumNtState, and takes advantage of the
preparations made by the preceding commit.  The macro is shrinking.

DEV-7145
2022-09-14 15:35:58 -04:00
Mike Gerwitz 1817659811 tamer: xir::parse::ele: Abstract child NT states in parent parser
This is in preparation for hoisting out the common states, as was done with
the Sum NT in a previous commit.

I also think that organizing states in this way is more clear.  The previous
embedding of the variants named after the NTs themselves was because the
parser was storing the child state within it, before the introduction of the
superstate trampoline.

DEV-7145
2022-09-14 14:47:54 -04:00
Mike Gerwitz d73a18d1a2 tamer: xir::parse::ele: Initial extraction of Sum NT state from macro
After introducing the superstate and trampoline some time ago, the Sum NT
states became fully generalized and can be hoisted out.

DEV-7145
2022-09-14 12:23:52 -04:00
Mike Gerwitz db3fd3f177 tamer: xir::parse::ele: Remove `unreachable!` in state transitions
This will instead fail at compile time.

DEV-7145
2022-09-14 10:00:10 -04:00
Mike Gerwitz a5c7067c68 tamer: xir::parse::ele: Remove NT `todo!` for state transition
Everything except for one state was already accounted for.  We can now have
confidence that the parser will never panic due to state transitions (beyond
legitimate error conditions).

There are some `unreachable!`s to contend with still.

DEV-7145
2022-09-14 09:41:53 -04:00
Mike Gerwitz 212ca06efe tamer: xir::parse: Extract and generalize NT errors
This is the same as the previous commits, but for non-sum NTs.

This also extracts errors into a separate module, which I had hoped to do in
a separate commit, but it's not worth separating them.  My _original_ reason
for doing so was debugging (I'll get into that below), but I had wanted to
trim down `ele.rs` anyway, since that mess is large and a lot to grok.

My debugging was trying to figure out why Rust was failing to derive
`PartialEq` on `NtError` because of `AttrParseError`.  As it turns out,
`AttrParseError::InvalidValue` was failing, thus the introduction of the
`PartialEq` trait bound on `AttrParseState::ValueError`.  Figuring this out
required implementing `PartialEq` myself without `derive` (well, using LSP,
which did all the work for me).

I'm not sure why this was not failing previously, which is a bit of a
concern, though perhaps in the context of the macro-expanded code, Rust was
able to properly resolve the types.

DEV-7145
2022-09-14 09:28:31 -04:00
Mike Gerwitz 5078bd8bda tamer: xir::parse::ele: Extract sum NT error from `ele_parse!`
The `ele_parse!` macro is a monstrosity, and expands into many different
identifiers.  The hope is that chipping away at things like this will not
only make the template easier to understand by framing portions of the
problem in terms of more traditional Rust code, but will also hopefully
reduce compile times by reducing the amount of code that is expanded by the
macro.

DEV-7145
2022-09-13 09:20:29 -04:00
Mike Gerwitz 0ac24baa87 build-aux/Makefile.am (bootstrap-if-necessary): New target
This introduces an order-only prerequisite `bootstrap-if-necessary` for the
generation of `suppliers.mk`.  Projects utilizing TAME as a dependency may
include a `bootstrap.mk` that overrides this target to trigger any
bootstrapping scripts that may be necessary due to toolchain updates.

DEV-7145
2022-09-07 11:18:42 -04:00
Mike Gerwitz 5edefde201 Makefile (bin): Target to build only binaries
Systems utilizing TAME as a build dependency are not interested in
everything else that gets built (tests and docs, primarily).

DEV-7145
2022-09-07 09:53:44 -04:00
Mike Gerwitz 1dfd5d89cb tame: bootstrap: Stop building after bootstrap
This was a relic of the old bootstrap system, where bootstrapping was in the
context of a parent project that utilized TAME (and so TAME needed to be
built).  But that doesn't make sense in the context of TAME itself, and what
_part_ of TAME should be built should be controlled by the project utilizing
it.

This is especially important now that TAMER builds are getting much longer
with the introduction of NIR and its parser-generator.

DEV-7145
2022-09-06 13:55:03 -04:00
Mike Gerwitz a712d8b279 .gitlab-ci.yml: Filter TAMER artifacts to binaries
The binaries are small, but the other data in target/ is huge (>1GiB on my
machine after numerous builds).

DEV-7145
2022-08-29 16:15:37 -04:00
Mike Gerwitz 419b24f251 tamer: Introduce NIR (accepting only)
This introduces NIR, but only as an accepting grammar; it doesn't yet emit
the NIR IR, beyond TODOs.

This modifies `tamec` to, while copying XIR, also attempt to lower NIR to
produce parser errors, if any.  It does not yet fail compilation, as I just
want to be cautious and observe that everything's working properly for a
little while as people use it, before I potentially break builds.

This is the culmination of months of supporting effort.  The NIR grammar is
derived from our existing TAME sources internally, which I use for now as a
test case until I introduce test cases directly into TAMER later on (I'd do
it now, if I hadn't spent so much time on this; I'll start introducing tests
as I begin emitting NIR tokens).  This is capable of fully parsing our
largest system with >900 packages, as well as `core`.

`tamec`'s lowering is a mess; that'll be cleaned up in future commits.  The
same can be said about `tameld`.

NIR's grammar has some initial documentation, but this will improve over
time as well.

The generated docs still need some improvement, too, especially with
generated identifiers; I just want to get this out here for testing.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz c420ab2730 tamer: xir::parse: Correct doc xrefs
These weren't causing problems until they were output as part of NIR (in a
separate module).

NIR is about to be committed.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 638a9c483b tamer: xir::parse::ele: Hide internal NT enum variants
The user never sees or interacts with these; they're macro-generated, and
distract from the useful information in the generated docs.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 2b33a45985 tamer: xir::parse::ele: Support NT docs
This just modifies the macro to proxy attributes to generated NTs so that
they can be documented.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 51728545f7 tamer: xir::parse::ele: Properly handle previous state transitions
This includes when on the last state / expecting a close.

Previously, there were a couple major issues:

  1. After parsing an NT, we can't allow preemption because we must emit a
     dead state so that we can remove the NT from the stack, otherwise
     they'll never close (until the parent does) and that results in
     unbounded stack growth for a lot of siblings.  Therefore, we cannot
     preempt on `Text`, which causes the NT to receive it, emit a dead
     state, transition away from the NT, and not accept another NT of the
     same type after `Text`.

  2. When encountering an unknown element, the error message stated that a
     closing tag was expected rather than one of the elements accepted by the
     final NT.

For #1, this was solved by allowing the parent to transition back to the NT
if it would have been matched by the previous NT.  A future change may
therefore allow us to remove repetition handling entirely and allow the
parent to deal with it (maybe).

For #2, the trouble is with the parser generator macro---we don't have a
good way of knowing the last NT, and the last NT may not even exist if none
was provided.  This solution is a compromise, after having tried and failed
at many others; I desperately need to move on, and this results in the
correct behavior and doesn't sacrifice performance.  But it can be done
better in the future.

It's also worth noting for #2 that the behavior isn't _entirely_ desirable,
but in practice it is mostly correct.  Specifically, if we encounter an
unknown token, we're going to blow through all NTs until the last one, which
will be forced to handle it.  After that, we cannot return to a previous NT,
and so we've forefitted the ability to parse anything that came before it.

NIR's grammar is such that sequences are rare and, if present, there's
really only ever two NTs, and so this awkward behavior will rarely cause
practical issues.  With that said, it ought to be improved in the future,
but let's wait to see if other parts of the lowering pipeline provide more
appropriate places to handle some of these things (even though it really
ought to be handled at the grammar level).

But I'm well out of time to spend on this.  I have to move on.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 2fcd0b35ae core: vector/cmatch/match-* (@const@): Remove
This removes the deprecated `@const@` argument in favor of shorthand
`@value@` constants, which were introduced long ago precisely to avoid
having to define separate `@const@` parameters for all of these templates.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 5ee0ddd064 core: list2typedef: Include proper package namespace prefixes
Required by yet-to-be-committed TAMER grammar.

DEV-7145
2022-08-29 15:52:03 -04:00
Mike Gerwitz 8a286878f6 core: numeric/round: Add missing `item/@desc`
This was caught by TAMER using a yet-to-be-committed NIR.

DEV-7145
2022-08-22 15:02:53 -04:00
Mike Gerwitz 93fb6e78e4 core: numeric/round: Remove c:value-of/@type
This is not valid and never was; TAME just didn't validate inside templates,
unlike TAMER.

DEV-7145
2022-08-22 15:02:53 -04:00
Mike Gerwitz 6269f8de6e core: Remove @keep
"keep" is an old feature that forced the linker to retain symbols that were
unused.  This was removed long ago in favor of having all linker roots
defined by the return map.

This also removes an old `@always`, which seems like a typo for
`when="always"` or something...not entirely sure.

DEV-7145
2022-08-22 15:02:53 -04:00
Mike Gerwitz acd0aea6a4 core: Remove @accumulate
Accumulators were an ancient TAME feature removed long ago during The Great
Refactoring (...okay, that part didn't fit the definition of a "refactor",
but that's technically what that's referring to).

TAMER will not accept it.

DEV-7145
2022-08-22 15:02:52 -04:00
Mike Gerwitz 3e1bf48a45 core: Remove __DATE_YEAR__
This has not been in use for years and it's time to go away---it is the only
thing in TAME that causes nondeterminism, at least that I'm immediately
aware of.  Perhaps I'll find something else while reimplementing TAME in
TAMER.

_This does not remove the compiler code to produce this._  If something
still needs `__DATE_YEAR__` (because it's really old), it can define this
value itself, and still utilize it until TAMER (which will not include it).

DEV-7145
2022-08-22 15:02:52 -04:00
Mike Gerwitz 7466ecbe8b tamer: xir::parse::ele: Accept missing child
`ele_parse!` was recently converted to accept zero-or-more for every NT to
simplify the parser-generator, since NIR isn't going to be able to
accurately determine whether child requirements are met anyway (because of
the template system).

This ensures that `Close` can be accepted when we're expecting an
element.  It also adds a test for a scenario that's causing me some trouble
in stashed code so that I can ensure that it doesn't break.

DEV-7145
2022-08-22 09:43:59 -04:00
Mike Gerwitz 9366c0c154 tamer: xir::parse::ele: Increase parser nesting depth
This sets the maximum depth to 64, which is still arbitrary, but
unfortunately the sum types introduce multiple levels of nesting, in
particular for template applications, so nested applications can result in a
fairly large stack.

I have various ideas to improve upon that---limited a bit in that repetition
as it is current implemented inhibits tail calls---but they're not worth
doing just yet relative to other priorities.  The impact of this change is
not significant.

DEV-7145
2022-08-18 16:16:45 -04:00
Mike Gerwitz abb2c80e22 tamer: xir::parse::ele: Always repeat
This removes support for configurable repetition.

What?  Why?

As it turns out, the complexity that repetition adds is quite significant
and is not worth the effort.  The truth is that NIR is going to have to
allow zero-or-more matches on virtually everything _anyway_ because template
application is allowed virtually anywhere---it is not possible to fully
statically analyze TAME's sources because templates can expand into just
about anything.  Given that, AIR (or something down the line) is going to
have to supply the necessary invariants instead.

It does suck, though, that this removes a lot of code that I fairly recently
wrote, and spent a decent amount of time on.  But it's important to know
when to cut your losses.

Perhaps I could have planned better, but deriving this whole system as been
quite the experiment.

DEV-7145
2022-08-18 15:19:40 -04:00
Mike Gerwitz 13d3c76a31 tamer: xir::parse::ele: Test to verify close after child recovery
Just want to be sure that we emit a closing object to match the emitted
opening one after recovery, otherwise the IR becomes unbalanced.

DEV-7145
2022-08-18 12:41:27 -04:00
Mike Gerwitz 955131217b tamer: xir::parse::ele: Attribute dead state recovery
If attributes fail to parse (e.g. missing required attribute) and parsing
reaches a dead state, this will recover by ignoring the entire element.  It
previously panicked with a TODO.

DEV-7145
2022-08-18 12:41:26 -04:00