Commit Graph

557 Commits (0ac24baa877352337c2c75f2622d297fb991e204)

Author SHA1 Message Date
Mike Gerwitz 419b24f251 tamer: Introduce NIR (accepting only)
This introduces NIR, but only as an accepting grammar; it doesn't yet emit
the NIR IR, beyond TODOs.

This modifies `tamec` to, while copying XIR, also attempt to lower NIR to
produce parser errors, if any.  It does not yet fail compilation, as I just
want to be cautious and observe that everything's working properly for a
little while as people use it, before I potentially break builds.

This is the culmination of months of supporting effort.  The NIR grammar is
derived from our existing TAME sources internally, which I use for now as a
test case until I introduce test cases directly into TAMER later on (I'd do
it now, if I hadn't spent so much time on this; I'll start introducing tests
as I begin emitting NIR tokens).  This is capable of fully parsing our
largest system with >900 packages, as well as `core`.

`tamec`'s lowering is a mess; that'll be cleaned up in future commits.  The
same can be said about `tameld`.

NIR's grammar has some initial documentation, but this will improve over
time as well.

The generated docs still need some improvement, too, especially with
generated identifiers; I just want to get this out here for testing.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz c420ab2730 tamer: xir::parse: Correct doc xrefs
These weren't causing problems until they were output as part of NIR (in a
separate module).

NIR is about to be committed.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 638a9c483b tamer: xir::parse::ele: Hide internal NT enum variants
The user never sees or interacts with these; they're macro-generated, and
distract from the useful information in the generated docs.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 2b33a45985 tamer: xir::parse::ele: Support NT docs
This just modifies the macro to proxy attributes to generated NTs so that
they can be documented.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 51728545f7 tamer: xir::parse::ele: Properly handle previous state transitions
This includes when on the last state / expecting a close.

Previously, there were a couple major issues:

  1. After parsing an NT, we can't allow preemption because we must emit a
     dead state so that we can remove the NT from the stack, otherwise
     they'll never close (until the parent does) and that results in
     unbounded stack growth for a lot of siblings.  Therefore, we cannot
     preempt on `Text`, which causes the NT to receive it, emit a dead
     state, transition away from the NT, and not accept another NT of the
     same type after `Text`.

  2. When encountering an unknown element, the error message stated that a
     closing tag was expected rather than one of the elements accepted by the
     final NT.

For #1, this was solved by allowing the parent to transition back to the NT
if it would have been matched by the previous NT.  A future change may
therefore allow us to remove repetition handling entirely and allow the
parent to deal with it (maybe).

For #2, the trouble is with the parser generator macro---we don't have a
good way of knowing the last NT, and the last NT may not even exist if none
was provided.  This solution is a compromise, after having tried and failed
at many others; I desperately need to move on, and this results in the
correct behavior and doesn't sacrifice performance.  But it can be done
better in the future.

It's also worth noting for #2 that the behavior isn't _entirely_ desirable,
but in practice it is mostly correct.  Specifically, if we encounter an
unknown token, we're going to blow through all NTs until the last one, which
will be forced to handle it.  After that, we cannot return to a previous NT,
and so we've forefitted the ability to parse anything that came before it.

NIR's grammar is such that sequences are rare and, if present, there's
really only ever two NTs, and so this awkward behavior will rarely cause
practical issues.  With that said, it ought to be improved in the future,
but let's wait to see if other parts of the lowering pipeline provide more
appropriate places to handle some of these things (even though it really
ought to be handled at the grammar level).

But I'm well out of time to spend on this.  I have to move on.

DEV-7145
2022-08-29 15:52:04 -04:00
Mike Gerwitz 7466ecbe8b tamer: xir::parse::ele: Accept missing child
`ele_parse!` was recently converted to accept zero-or-more for every NT to
simplify the parser-generator, since NIR isn't going to be able to
accurately determine whether child requirements are met anyway (because of
the template system).

This ensures that `Close` can be accepted when we're expecting an
element.  It also adds a test for a scenario that's causing me some trouble
in stashed code so that I can ensure that it doesn't break.

DEV-7145
2022-08-22 09:43:59 -04:00
Mike Gerwitz 9366c0c154 tamer: xir::parse::ele: Increase parser nesting depth
This sets the maximum depth to 64, which is still arbitrary, but
unfortunately the sum types introduce multiple levels of nesting, in
particular for template applications, so nested applications can result in a
fairly large stack.

I have various ideas to improve upon that---limited a bit in that repetition
as it is current implemented inhibits tail calls---but they're not worth
doing just yet relative to other priorities.  The impact of this change is
not significant.

DEV-7145
2022-08-18 16:16:45 -04:00
Mike Gerwitz abb2c80e22 tamer: xir::parse::ele: Always repeat
This removes support for configurable repetition.

What?  Why?

As it turns out, the complexity that repetition adds is quite significant
and is not worth the effort.  The truth is that NIR is going to have to
allow zero-or-more matches on virtually everything _anyway_ because template
application is allowed virtually anywhere---it is not possible to fully
statically analyze TAME's sources because templates can expand into just
about anything.  Given that, AIR (or something down the line) is going to
have to supply the necessary invariants instead.

It does suck, though, that this removes a lot of code that I fairly recently
wrote, and spent a decent amount of time on.  But it's important to know
when to cut your losses.

Perhaps I could have planned better, but deriving this whole system as been
quite the experiment.

DEV-7145
2022-08-18 15:19:40 -04:00
Mike Gerwitz 13d3c76a31 tamer: xir::parse::ele: Test to verify close after child recovery
Just want to be sure that we emit a closing object to match the emitted
opening one after recovery, otherwise the IR becomes unbalanced.

DEV-7145
2022-08-18 12:41:27 -04:00
Mike Gerwitz 955131217b tamer: xir::parse::ele: Attribute dead state recovery
If attributes fail to parse (e.g. missing required attribute) and parsing
reaches a dead state, this will recover by ignoring the entire element.  It
previously panicked with a TODO.

DEV-7145
2022-08-18 12:41:26 -04:00
Mike Gerwitz 77fd92bbb2 tamer: xir::parse::ele: Remove `_` suffix from error variants
These were initially used to prevent conflicts with generated variants, but
we are no longer generating such variants since they're being jumped to via
the trampoline.

DEV-7145
2022-08-17 14:58:54 -04:00
Mike Gerwitz b31ebc00a7 tamer: xir::parse::ele: Handle Close when expecting Open
I'm starting to clean up some TODOs, and this was a glaring one causing
panics when encountered.  The recovery for this is simple, because we have
no choice: just stop parsing; leave it to the next lowering operation(s) to
complain that we didn't provide what was necessary.  They'll have to,
anyway, since templates mean that NIR cannot ever have enough information to
guarantee that a document is well-formed, relative to what would expand from
the template.

DEV-7145
2022-08-17 14:49:34 -04:00
Mike Gerwitz 4c86c5b63c tamer: xir::parse::ele: Support nested Sum NTs
This allows for a construction like this:

```
ele_parse! {
  [...]

  StmtX := QN_X {
    [...]
  };

  StmtY := QN_Y {
    [...]
  };

  ExprA := QN_A {
    [...]
  };

  ExprB := QN_B {
    [...]
  };

  Expr := (A | B);
  Stmt := (StmtX | StmtY);

  // This previously was not allowed:
  StmtOrExpr := (Stmt | Expr);
}
```

There were initially two barriers to doing so:

  1. Efficiently matching; and
  2. Outputting diagnostic information about the union of all expected
     elements.

The first was previously resolved with the introduction of `NT::matches`,
which is macro-expanded in a way that Rust will be able to optimize a
bit.  Worst case, it's effectively a linear search, but our Sum NTs are not
that deep in practice, so I don't expect that to be a concern.

The concern that I was trying to avoid was heap-allocated `NodeMatcher`s to
avoid recursive data structures, since that would have put heap access in a
very hot code path, which is not an option.

That left problem #2, which ended up being the harder problem.  The solution
was detailed in the previous commit, so you should look there, but it
amounts to being able to format individual entries as if they were a part
of a list by making them a function of not just the matcher itself, but also
the number of items in (recursively) the sum type and the position of the
matcher relative to that list.  The list length is easily
computed (recursively) at compile-time using `const`
functions (`NT::matches_n`).

And with that, NIR can be abstracted in sane ways using Sum NTs without a
bunch of duplication that would have been a maintenance burden and an
inevitable source of bugs (from having to duplicate NT references).

DEV-7145
2022-08-17 10:44:53 -04:00
Mike Gerwitz fd3184c795 tamer: fmt (ListDisplayWrapper::fmt_nth): List display without a slice
This exposes the internal rendering of `ListDisplayWrapper::fmt` such that
we can output a list without actually creating a list.  This is used in an
upcoming change for =ele_parse!= so that Sum NTs can render the union of all
the QNames that their constituent NTs match on, recursively, as a single
list, without having to create an ephemeral collection only for display.

If Rust supports const functions for arrays/Vecs in the future, we could
generate this at compile-time, if we were okay with the (small) cost, but
this solution seems just fine.  But output may be even _more_ performant
since they'd all be adjacent in memory.

This is used in these secenarios:

  1. Diagnostic messages;
  2. Error messages (overlaps with #1); and
  3. `Display::fmt` of the `ParseState`s themselves.

The reason that we want this to be reasonably performant is because #3
results in a _lot_ of output---easily GiB of output depending on what is
being traced.  Adding heap allocations to this would make it even slower,
since a description is generated for each individual trace.

Anyway, this is a fairly simple solution, albeit a little bit less clear,
and only came after I had tried a number of other different approaches
related to recursively constructing QName lists at compile time; they
weren't worth the effort when this was so easy to do.

DEV-7145
2022-08-17 10:44:28 -04:00
Mike Gerwitz 6b29479fd6 tamer: xir::fmt (DisplayFn): New fn wrapper
See the docblock for a description.  This is used in an upcoming commit for
=ele_parse!=.

DEV-7145
2022-08-17 10:01:47 -04:00
Mike Gerwitz 4177b8ed71 tamer: xir::parse::ele: Streaming attribute parsing
This allows using a `[attr]` special form to stream attributes as they are
encountered rather than aggregating a static attribute list.  This is
necessary in particular for short-hand template application and short-hand
function application, since the attribute names are derived from template
and function parameter lists, which are runtime values.

The syntax for this is a bit odd since there's a semi-useless and confusing
`@ {} => obj` still, but this is only going to be used by a couple of NTs
and it's not worth the time to clean this up, given the rather significant
macro complexity already.

DEV-7145
2022-08-16 23:06:38 -04:00
Mike Gerwitz 43c64babb0 tamer: xir::parse::ele: Superstate element preemption
This uses the same mechanism that was introduced for handling `Text` nodes
in mixed content, allowing for arbitrary element `Open` matches for
preemption by the superstate.

This will be used to allow for template expansion virtually
anywhere.  Unlike the existing TAME, it'll even allow for it at the root,
though whether that's ultimately permitted is really depending on how I
approach template expansion; it may fail during a later lowering operation.

This is interesting because this approach is only possible because of the
CPS-style trampoline implementation.  Previously, with the composition-based
approach, each and every parser would have to perform this check, like we
had to previously with `Text` nodes.

As usual, this is still adding to the mess a bit, and it'll need some future
cleanup.

DEV-7145
2022-08-16 15:47:41 -04:00
Mike Gerwitz 6f53c0971b tamer: xir::parse::ele: Superstate text node preemption
This introduces the concept of superstate node preemption generally, which I
hope to use for template application as well, since templates can appear in
essentially any (syntatically valid, for XML) position.

This implements mixed content handling by defining the mapping on the
superstate itself, which really simplifies the problem but foregoes
fine-grained text handling.  I had hoped to avoid that, but oh well.

This pushes the responsibility of whether text is semantically valid at that
position to NIR->AIR lowering (which we're not transition to yet), which is
really the better place for it anyway, since this is just the grammar.  The
lowering to AIR will need to validate anyway given that template expansion
happens after NIR.

Moving on!

DEV-7145
2022-08-16 12:26:24 -04:00
Mike Gerwitz 65b42022f0 tamer: xir::st: Prefix all preproc-namespaced constants with `QN_P_`
I had previously avoided this to keep names more concise, but now it's
ambiguous with parsing actual TAME sources.

DEV-7145
2022-08-15 13:00:10 -04:00
Mike Gerwitz 13641e1812 tamer: diagnose::report: `int_log` feature: {=>i}log10
https://github.com/rust-lang/rust/pull/100332

The above MR replaces `log10` and friends with `ilog10`; this is the first
time an unstable feature bit us in a substantially backwards-incompatible
way that's a pain to deal with.

Fortunately, I'm just not going to deal with it: this is used with the
diagnostic system, which isn't yet used by our projects (outside of me
testing), and so those builds shouldn't fail before people upgrade.

This is now pending stabalization with the new name, so hopefully we're good
now:

  https://github.com/rust-lang/rust/issues/70887#issuecomment-1210602692
2022-08-12 16:42:30 -04:00
Mike Gerwitz 2a36bc4210 tamer: (explicit_generic_args_with_impl_trait): Remove unstable feature flag
This was stabalized in Rust 1.63.  I was waiting to be sure our build
servers were updated properly before removing this (and they were, long
ago).
2022-08-12 16:42:30 -04:00
Mike Gerwitz ed8a2ce28a tamer: xir::parse::ele: Superstate not to accept early EOF
This was accepting an early EOF when the active child `ParseState` was in an
accepting state, because it was not ensuring that anything on the stack was
also accepting.

Ideally, there should be nothing on the stack, and hopefully in the future
that's what happens.  But with how things are today, it's important that, if
anything is on the stack, it is accepting.

Since `is_accepting` on the superstate is only called during finalization,
and because the check terminates early, and because the stack practically
speaking will only have a couple things on it max (unless we're in tail
position in a deeply nested tree, without TCO [yet]), this shouldn't be an
expensive check.

Implementing this did require that we expose `Context` to `is_accepting`,
which I had hoped to avoid having to do, but here we are.

DEV-7145
2022-08-12 00:47:15 -04:00
Mike Gerwitz a4419413fb tamer: parse::trace: Include context
This is something that I had apparently forgotten to do, but is now useful
in debugging `ele_parse!` issues with the trampoline.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz 54d8348e95 tamer: Add `--quiet` flag to `make check` (`cargo test`)
I wonder when this option was introduced, unless I never saw it because it
is called "quiet".  But this is what I always wanted (and how I write the
output for my own tools, like progtest in this repo); the output has long
gotten far too large.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz 22a9596cf4 tamer: xir::parse::ele: Hoist whitespace/comment handling to superstate
All child parsers do the same thing, so this simplifies things.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz f8a9e952e5 tamer: xir::parse::ele: Correct handling of sum dead state post-recovery
Along with this change we also had to change how we handle dead states in
the superstate.  So there were two problems here:

  1. Sum states were not yielding a dead state after recovery, which meant
     that parsing was unable to continue (we still have a `todo!`); and
  2. The superstate considered it an error when there was nothing left on
     the stack, because I assumed that ought not happen.

Regarding #2---it _shouldn't_ happen, _unless_ we have extra input after we
have completed parsing.  Which happens to be the case for this test case,
but more importantly, we shouldn't be panicing with errors about TAMER bugs
if somebody puts extra input after a closing root tag in a source file.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz b95ec5a9d8 tamer: xir::parse::ele: Adjust diagnostic display of expected element list
This does two things:

  1. Places the expected list on a separate help line as a footnote where
     it'll be a bit more tolerable when it inevitably overflows the terminal
     width in certain contexts (we may wrap in the future); and
  2. Removes angled brackets from the element names so that they (a) better
     correspond with the span which highlights only the element name and (b)
     do not imply that the elements take no attributes.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz 67ee914505 tamer: xir::parse::ele: Store matching QName on NS match
When we match a QName against a namespace, we ought to store the matching
QName to use (a) in error messages and (b) to make available as a
binding.  The former is necessary for sensible errors (rather than saying
that it's e.g. expecting a closing `t:*`) and the latter is necessary for
e.g. getting the template name out of `t:foo`.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz 8cb03d8d16 tamer: xir::parse::ele: Initial namespace prefix matching support
This allows matching on a namespace prefix by providing a `Prefix` instead
of a `QName`.  This works, but is missing a couple notable things (and
possibly more):

  1. Tracking the QName that is _actually_ matched so that it can be used in
     messages stating what the expected closing tag is; and
  2. Making that QName available via a binding.

This will be used to match on `t:*` in NIR.  If you're wondering how
attribute parsing is supposed to work with that (of course you're wondering
that, random person reading this)---that'll have to work differently for
those matches, since template shorthand application contains argument names
as attributes.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz f9fe4aa13b tamer: xir::st: Static namespace prefixes (c and t)
In particular, `t:*` will be recognized by NIR for short-hand template
application.  These will be utilized in an upcoming commit.

DEV-7145
2022-08-12 00:47:14 -04:00
Mike Gerwitz 88fa0688fa tamer: xir::parse::ele: Abstract node matching
This introduces `NodeMatcher`, with the intent of introducing wildcard QName
matches for e.g. `t:*` nodes.  It's not yet clear if I'll expand this to
support text nodes yet, or if I'll convert text nodes into elements to
re-use the existing system (which I had initially planned on doing, but
didn't because of the work and expense (token expansion) involved in the
conversion).

DEV-7145
2022-08-12 00:47:13 -04:00
Mike Gerwitz 7b9bc9e108 tamer: xir::parse::ele: Ignore Text nodes for now
I need to move on, and there are (a) a couple different ways to proceed that
I want to mull over and (b) upcoming changes that may influence my decision
one way or another.

DEV-7145
2022-08-12 00:47:12 -04:00
Mike Gerwitz 4aaf91a9e7 tamer: xir::parse::ele: Un-nest child parser errors
This will utilize the superstate's error object in place of nested errors,
which was the result of the previous composition-based delegation.

As you can see, all we had to do was remove the special handling of these
errors; the existing delegation setup continues to handle the types properly
with no change.  The composition continues to work for `*Attr_`.

The alternative was to box inner errors, since they're far from the hot code
path, but that's clearly unnecessary.

To be clear: this is necessary to allow for recursive grammars in
`ele_parse` without creating recursive data structures in Rust.

DEV-7145
2022-08-10 11:46:54 -04:00
Mike Gerwitz adf7baf115 tamer: xir::parse::ele: Handle comments like whitespace
Comments ought not have any more semantic meaning than whitespace.  Other
languages may have conventions that allow for various types of things in
comments, like annotations, but those are symptoms of language
limitations---we control the source language here.

DEV-7145
2022-08-10 11:46:54 -04:00
Mike Gerwitz 15e04d63e2 tamer: xir::parse::ele: Transition trampoline
This properly integrates the trampoline into `ele_parse!`.  The
implementation leaves some TODOs, most notably broken mixed text handling
since we can no longer intercept those tokens before passing to the
child.  That is temporarily marked as incomplete; see a future commit.

The introduced test `ParseState`s were to help me reason about the system
intuitively as I struggled to track down some type errors in the monstrosity
that is `ele_parse!`.  It will fail to compile if those invariants are
violated.  (In the end, the problems were pretty simple to resolve, and the
struggle was the type system doing its job in telling me that I needed to
step back and try to reason about the problem again until it was intuitive.)

This keeps around the NT states for now, which are quickly used to
transition to the next NT state, like a couple of bounces on a trampoline:

  NT -> Dead -> Parent -> Next NT

This could be optimized in the future, if it's worth doing.

This also makes no attempt to implement tail calls; that would have to come
after fixing mixed content and really isn't worth the added complexity
now.  I (desperately) need to move on, and still have a bunch of cleanup to
do.

I had hoped for a smaller commit, but that was too difficult to do with all
the types involved.

DEV-7145
2022-08-10 11:46:45 -04:00
Mike Gerwitz 233fa7de6a tamer: diagnose::panic: New module
This change introduces diagnostic messages for panics.  The intent is to be
able to use panics in situations where it is either not possible to or not
worth the time to recover from errors and ensure a consistent/sensible
system state.  In those situations, we still ought to be able to provide the
user with useful information to attempt to get unstuck, since the error is
surely in response to some particular input, and maybe that input can be
tweaked to work around the problem.

Ideally, invalid states are avoided using the type system and statically
verified at compile-time.  But this is not always possible, or in some cases
may be way more effort or cause way more code complexity than is worth,
given the unliklihood of the error occurring.

With that said, it's been interesting, over the past >10y that TAME has
existed, seeing how unlikely errors do sometimes pop up many years after
they were written.  It's also interesting to have my intuition of what is
"unlikely" challenged, but hopefully it holds generally.

DEV-7145
2022-08-09 15:20:37 -04:00
Mike Gerwitz 454b7a163f tamer: xir::parse::ele: Move repeat configuration out of Context
I had previously used `Context` to hold the parser configuration for
repetition, since that was the easier option.  But I now want to utilize the
`Context` for a stack for the superstate trampoline, and I don't want to
have to deal with the awkwardness of the repetition in doing so, since it
requires that the configuration be created during delegation, rather than
just being passed through to all child parsers.

This adds to a mess that needs cleaning up, but I'll do that after
everything is working.

DEV-7145
2022-08-08 15:23:55 -04:00
Mike Gerwitz 6bc872eb38 tamer: xir::parse::ele: Generate superstate
And here's the thing that I've been dreading, partly because of the
`macro_rules` issues involved.  But, it's not too terrible.

This module was already large and complex, and this just adds to it---it's
in need of refactoring, but I want to be sure it's fully working and capable
of handling NIR before I go spending time refactoring only to undo it.

_This does not yet use trampolining in place of the call stack._  That'll
come next; I just wanted to get the macro updated, the superstate generated,
and tests passing.  This does convert into the
superstate (`ParseState::Super`), but then converts back to the original
`ParseState` for BC with the existing composition-based delegation.  That
will go away and will then use the equivalent of CPS, using the
superstate+`Parser` as a trampoline.  This will require an explicit stack
via `Context`, like XIRF.  And it will allow for tail calls, with respect to
parser delegation, if I decide it's worth doing.

The root problem is that source XML requires recursive parsing (for
expressions and statements like `<section>`), which results in recursive
data structures (`ParseState` enum variants).  Resolving this with boxing is
not appropriate, because that puts heap indirection in an extremely hot code
path, and may also inhibit the aggressive optimizations that I need Rust to
perform to optimize away the majority of the lowering pipeline.

Once this is sorted out, this should be the last big thing for the
parser.  This unfortunately has been a nagging and looming issue for months,
that I was hoping to avoid, and in retrospect that was naive.

DEV-7145
2022-08-08 15:23:55 -04:00
Mike Gerwitz 53a689741b tamer: parse::state::ParseState::Super: Superstate concept
I'm disappointed that I keep having to implement features that I had hoped
to avoid implementing.

This introduces a "superstate" feature, which is intended really just to be
a sum type that is able to delegate to stitched `ParseState`s.  This then
allows a `ParseState` to transition directly to another `ParseState` and
have the parent `ParseState` handle the delegation---a trampoline.

This issue naturally arises out of the recursive nature of parsing a TAME
XML document, where certain statements can be nested (like `<section>`), and
where expressions can be nested.  I had gotten away with composition-based
delegation for now because `xmlo` headers do not have such nesting.

The composition-based approach falls flat for recursive structures.  The
typical naive solution is boxing, which I cannot do, because not only is
this on an extremely hot code path, but I require that Rust be able to
deeply introspect and optimize away the lowering pipeline as much as
possible.

Many months ago, I figured that such a solution would require a trampoline,
as it typically does in stack-based languages, but I was hoping to avoid
it.  Well, no longer; let's just get on with it.

This intends to implement trampolining in a `ParseState` that serves as that
sum type, rather than introducing it as yet another feature to `Parser`; the
latter would provide a more convenient API, but it would continue to bloat
`Parser` itself.  Right now, only the element parser generator will require
use of this, so if it's needed beyond that, then I'll debate whether it's
worth providing a better abstraction.  For now, the intent will be to use
the `Context` to store a stack that it can pop off of to restore the
previous `ParseState` before delegation.

DEV-7145
2022-08-08 15:23:54 -04:00
Mike Gerwitz 7a5f731cac tamer: tameld: XIRF nesting 64=>4
Since we'll never be reading past the header, this is all that is needed.

If in the future this is violated, XIRF will cause a nice diagnostic error
displaying precisely what opening tag caused the increased level of nesting,
which will aid in debugging and allow us to determine if it ought to be
increased.  Here's an example, if I set the max to `3`:

  error: maximum XML element nesting depth of `3` exceeded
     --> /home/.../foo.xmlo:261:10
      |
  261 |          <preproc:sym-ref name=":_vproduct:vector_a"/>
      |          ^^^^^^^^^^^^^^^^ error: this opening tag increases the level of nesting past the limit of 3

Of course, the longer-term goal is to do away with `xmlo` entirely.

This had no (perceivable via `/usr/bin/time -v`, at least) impact on memory
or CPU time.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 77efefe680 tamer: xir::attr::parse: Better parser state descriptions
The attribute name was neither quoted nor `@`-prefixed.  (I noticed this in
the traces.)

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 2d117a4864 tamer: xir::parse::ele: Mixed content parsing
"Mixed content" is the XML term representing element nodes mixed with text
nodes.  For example, `foo <strong>bar</strong> baz` is mixed.

TAME supports text nodes as documentation, intended to be in a literate
style but never fully realized.  In any case, we need to permit them, and I
wanted to do more than just ignore the nodes.

This takes a different approach than typical parser delegation---it has the
parent parser _preempt_ the child by intercepting text before delegation
takes place, rather than having the child reject the token (or possibly
interpret it itself!) and have to handle an error or dead state.

And while this makes it more confusing in terms of state machine stitching,
it does make sense, in the sense that the parent parser is really what
"owns" the text node---the parser is delegating _element_ parsing only, take
asserts authority when necessary to take back control where it shouldn't be
delegated.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 8779abe2bb tamer: xir::flat: Expose depth for all node-related tokens
Previously a `Depth` was provided only for `Open` and `Close`.  This depth
information, for example, will be used by NIR to quickly determine whether a
given parser ought to assert ownership of a text/comment token rather than
delegating it.

This involved modifying a number of test cases, but it's worth repeating in
these commits that this is intentional---I've been bit in the past using
`..` in contexts where I really do want to know if variant fields change so
that I can consider whether and how that change may affect the code
utilizing that variant.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz b3c0bdc786 tamer: xir::parse::ele: Ignore whitespace around elements
Recent changes regarding whitespace were all to support this change (though
it was also needed for XIRF, pre- and post-root).

Now I'll have to conted with how I want to handle text nodes in various
circumstances, in terms of `ele_parse!`.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 8f3301431c tamer: span::dummy: New module to hold DUMMY_SPAN and derivatives
Various DUMMY_SPAN-derived spans are used by many test cases, so this
finally extracts them---something I've been meaning to do for some time.

This also places DUMMY_SPAN behind a `cfg(test)` directive to ensure that it
is _only_ used in tests; UNKNOWN_SPAN should be used when a span is actually
unknown, which may also be the case during development.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 0edb21429d tamer: parse::error: Describe unexpected token of input
When Parser has a unhandled dead state and fails due to an unexpected token
of input, we should display what we interpreted that token as.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 18803ea576 tamer: xir: Format tokens without tt quotes
Whether or not quoting is appropriate depends on context, and that parent
context is already performing the quoting.  For example:

  error: expected `</rater>`, but found `<import>`
    --> /home/[...]/foo.xml:2:1
     |
   2 | <rater xmlns="http://www.lovullo.com/rater"
     | ------ note: element starts here

    --> /home/[...]/foo.xml:7:3
     |
   7 |   <import package="/rater/core/base" />
     |   ^^^^^^^ error: expected `</rater>`

In these cases (obviously I'm still working on the parser, since this is
nonsense), the parser is responsible for quoting the token "<import>".

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 8778976018 tamer: xir::flat: Ignore whitespace both before and after root
DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 4f2b27f944 tamer: xir: Attribute error formatting/typo fixes
There were two problem errors: one showing "element element" and one showing
the value along with the name of the attribute.

The change for `<Attr as Display>::fmt` is debatable.  I'm going to do this
for now (only show `@name`) and adjust later if necessary.

I'll need to go use `crate::fmt` consistently in previously-existing format
strings at some point, too.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 41b41e02c1 tamer: Xirf::Text refinement
This teaches XIRF to optionally refine Text into RefinedText, which
determines whether the given SymbolId represents entirely whitespace.

This is something I've been putting off for some time, but now that I'm
parsing source language for NIR, it is necessary, in that we can only permit
whitespace Text nodes in certain contexts.

The idea is to capture the most common whitespace as preinterned
symbols.  Note that this heuristic ought to be determined from scanning a
codebase, which I haven't done yet; this is just an initial list.

The fallback is to look up the string associated with the SymbolId and
perform a linear scan, aborting on the first non-whitespace character.  This
combination of checks should be sufficiently performant for now considering
that this is only being run on source files, which really are not all that
large.  (They become large when template-expanded.)  I'll optimize further
if I notice it show up during profiling.

This also frees XIR itself from being concerned by Whitespace.  Initially I
had used quick-xml's whitespace trimming, but it messed up my span
calculations, and those were a pain in the ass to implement to begin with,
since I had to resort to pointer arithmetic.  I'd rather avoid tweaking it.

tameld will not check for whitespace, since it's not important---xmlo files,
if malformed, are the fault of the compiler; we can ignore text nodes except
in the context of code fragments, where they are never whitespace (unless
that's also a compiler bug).

Onward and yonward.

DEV-7145
2022-08-01 15:01:37 -04:00