Commit Graph

1258 Commits (8948452b7112924fbbf66653eefb6b789ff3c48b)

Author SHA1 Message Date
Mike Gerwitz 8948452b71 tamer: asg::ident::Dim: Narrow type
This matches xmlo::Dim, and could be the same thing, if we can find a home
for it in the future; it's not worth creating such a home right now when I'm
not yet sure what else ought to live there; the duplication may be fine.

The conversion from xmlo needs to be moved, and `Dim` is going to be used
for more than just identifiers (expressions will have type inference
performed).

DEV-11864
2022-05-19 09:32:43 -04:00
Mike Gerwitz 263cb68380 tamer: parse: Persistent context
This allows retrieving and providing a context to a `Parser`.  This is
intended for use with an aggregating parser, in particular to construct the
ASG and return it.

This is a component of a change that replaces `asg_builder` with a
`Parser`-based lowering into the ASG, but there are still changes that need
to be made to simplify things and complete its integration.

DEV-11864
2022-05-18 16:15:09 -04:00
Mike Gerwitz 001499d921 tamer: parse::ParseError: Remove Eq trait bound
Just as in other commits, since it's an unnecessary limitation.

DEV-11864
2022-05-18 16:06:22 -04:00
Mike Gerwitz 3e277270a7 tamer: asg: Track roots on graph
Previously, since the graph contained only identifiers, discovered roots
were stored in a separate vector and exposed to the caller.  This not only
leaked details, but added complexity; this was left over from the
refactoring of the proof-of-concept linker some time ago.

This moves the root management into the ASG itself, mostly, with one item
being left over for now in the asg_builder (eligibility classifications).

There are two roots that were added automatically:

  - __yield
  - __worksheet

The former has been removed and is now expected to be explicitly mapped in
the return map, which is now enforced with an extern in `core/base`.  This
is still special, in the sense that it is explicitly referenced by the
generated code, but there's nothing inherently special about it and I'll
continue to generalize it into oblivion in the future, such that the final
yield is just a convention.

`__worksheet` is the only symbol of type `IdentKind::Worksheet`, and so that
was generalized just as the meta and map entries were.

The goal in the future will be to have this more under the control of the
source language, and to consolodate individual roots under packages, so that
the _actual_ roots are few.

As far as the actual ASG goes: this introduces a single root node that is
used as the sole reference for reachability analysis and topological
sorting.  The edges of that root node replace the vector that was removed.

DEV-11864
2022-05-17 10:42:05 -04:00
Mike Gerwitz 5a866f7735 core/base (___yield): New extern
Rather than having the linker add this symbol opaquely, let's remove the
special case and generalize it.  There's nothing special about yield, except
historical precedent.

Systems can explicitly add it as a root in a common return map.

DEV-11864
2022-05-16 15:07:37 -04:00
Mike Gerwitz 34eb994a0d tamer: asg::Asg::set_fragment: {ObjectRef=>SymbolId}
In the actual implementation (outside of tests), this is always looking up
before adding the symbol.  This will simplify the API, while still retaining
errors, since the identifier will fail the state transition if the
identifier did not exist before attempting to set a fragment.  So while this
is slower in microbenchmarks, this has no effect on real-world performance.

Further, I'm refactoring toward a streaming ASG aggregation, which is a lot
easier if we do not need to perform lookups in a separate step from the
ASG's primitives.

DEV-11864
2022-05-16 13:14:27 -04:00
Mike Gerwitz c49d87976d tamer: parse::Token: Remove Eq trait bound
`PartialEq` remains, and is all that is needed.  See previous commit
regarding the removal of this same bound from `Context`.

This can be re-added if it ends up actually being necessary.  But Tokens are
ephemeral and used only in lowering pipelines, using pattern matching.

DEV-11864
2022-05-16 10:05:14 -04:00
Mike Gerwitz d87006391e tamer: asg::object: Remove IdentObjectState, IdentObjectData
These traits are no longer necessary now that I'm using concrete types; they
just add unnecessary noise and confusion as I attempt to further refactor.

Don't abstract prematurely.

DEV-11864
2022-05-12 16:31:36 -04:00
Mike Gerwitz 3748762d31 tamer: asg::graph::Asg: Remove type parameter O
This removes the generic on the Asg (which was formerly BaseAsg),
hard-coding `IdentObject`, which will further evolve.  This makes the IR an
actual concrete IR rather than an abstract data structure.

These tests bring me back a bit, since they were written as I was still
becoming familiar with Rust.

DEV-11864
2022-05-12 15:46:17 -04:00
Mike Gerwitz 1114edbc6e rater/tame: Remove circular symlink
This was added long ago to maintain BC in some bizarre situation, and I had
forgotten about it, but it's causing problems with lsp-mode in Emacs.
2022-05-12 14:32:24 -04:00
Mike Gerwitz f2c5443176 tamer: asg: Remove generic Asg, rename {Base=>}Asg
This is the beginning of an incremental refactoring to remove generics, to
simplify the ASG.  When I initially wrote the linker, I wasn't sure what
direction I was going in, but I was also negatively influenced by more
traditional approaches to both design and unit testing.

If we're going to call the ASG an IR, then it needs to be one---if the core
of the IR is generic, then it's more like an abstract data structure than
anything.  We can abstract around the IR to slice it up into components that
are a little easier to reason about and understand how responsibilities are
segregated.

DEV-11864
2022-05-11 16:47:13 -04:00
Mike Gerwitz 0493e68cb3 tamer: parse::ParseState::Context: Add missing comment
DEV-11864
2022-05-10 11:06:22 -04:00
Mike Gerwitz 0ef0d2b553 tamer: parse::ParseState:Error: Relax Eq trait bound
This is unnecessarily restrictive, since we do not require anything further
than `PartialEq` for the situations where we care about equality (tests).

DEV-11864
2022-05-06 15:28:47 -04:00
Mike Gerwitz 9f990e19e9 tamer: parse::ParseState::Context: Remove Default trait bound
This is too restrictive, especially for parsers that fold into something,
like the ASG, which may exist prior to invoking the parser.

This moves the trait bound to the functions that actually need it.  Those
obviously cannot be used if the Context does not implement `Default`, but
I'll provide alternative conveniences.

DEV-11864
2022-05-05 15:55:04 -04:00
Mike Gerwitz ba9f429ee7 tamer: obj::xmlo::{XmloEvent=>XmloToken}
The original "event" name was based on quick-xml's `Event`.  This
terminology shift is more closely matched with the new parsing system.

DEV-11864
2022-05-05 12:25:59 -04:00
Mike Gerwitz 0d999b56cd src/current/summary.xsl: Correct invalid UTF-8 sequence
This broke when encoding was set to UTF-8 on this file.
2022-05-04 11:11:02 -04:00
Mike Gerwitz 2954c591a1 src/current/include/preproc/symtable: Remove extern @dtype check
I attempted to resolve an error previously, and I thought I had, but
apparently some symbols acquire a @dtype at some point in the process, or
lose it.  Regardless, I have no interest in debugging or resolving this
mess, since it's going away.

The linker ensures that externs match, so while this could potentially allow
conflicting imports within a package (unlikely, given that extern templates
are recommended), it still will not resolve with a conflicting concrete
implementation.  I'm not worried.

DEV-1036
2022-05-04 10:50:14 -04:00
Mike Gerwitz 0281dfdf0d tamer: Remove wip-frontends feature flag
We want the new system to be used so that we can start catching any problems
that may arise.  Further changes will be flagged as necessary.

DEV-10936
2022-05-04 09:37:10 -04:00
Mike Gerwitz 43c99cb61a src/current/include/preproc/symtable.xsl: Treat mutual missing extern @dtype as match
Extern resolution has apparently been failing for quite some time, resulting
in `preproc:error` nodes in the _symbol table_ of return maps.  This was
caught by the new xmlo parser, which does not ignore nodes it does not care
about.

The failure was caused by missing `@dtype`---the externs did in fact match,
and if they did not, then the linker would have failed.

This doesn't modify the map compiler to properly detect these, because
this compiler is going away in the hopefully-near future, and the problems
will now be caught, though in a very unideal way (as a parse error during
xmlo reading).

DEV-10936
2022-05-04 09:29:29 -04:00
Mike Gerwitz 602cec5560 src/current/compiler/map.xsl: Omit preproc:from from retmap symbols
preproc:sym/preproc:from is used for generating `knownFields` using the
_input_ map, so this has no use for return map values; the map still
produces edges to its dependencies.

The issue is that there are return map entries in some of our systems that
are producing multiple `preproc:from`, but I somewhat-recently modified the
system to support only a single map, to remove dynamic allocation.  This
resolves that problem.

With that said, `knownFields` was created for Liza to know when the
classifier ought to be invoked, to save time.  Back when it was first
introduced ~10y ago, this provided significant savings, however the
structure of our system now is such that nearly every single field invokes
the classifier.

Furthermore, these details should remain encapsulated; if we wanted to make
that determination, we should be provided with a delta, which we could also
use to do incremental classification in the future, if there's an ROI there
after other improvements have been made.

So, eventually, preproc:sym/preproc:from will go away entirely.

DEV-10936
2022-05-04 09:26:18 -04:00
Mike Gerwitz 1ad2fb1dc8 Copyright year update 2022
RSG (Ryan Specialty Group) recently announced a rename to Ryan Specialty (no
"Group"), but I'm not sure if the legal name has been changed yet or not, so
I'll wait on that.
2022-05-03 14:14:29 -04:00
Mike Gerwitz 34fcd19cd0 tamer: obj::xmlo::reader: Replace todo! with error
These are no longer TODOs---they represent invalid tokens.

I'm going to put effort into providing further context with the diagnostic
system [right now] because these are internal errors caused by either
miscompilation or an incomplete reader.

DEV-10936
2022-05-03 09:19:47 -04:00
Mike Gerwitz c4828e7e7a tame: src/current/compiler/worksheet: Place fragments in header
The new xmlo parser was failing on a worksheet xmlo file because fragments
were not properly placed within the header.

This was a change made when tameld was introduced so that we could stop
reading xmlo files early.

DEV-10936
2022-05-03 09:11:13 -04:00
Mike Gerwitz 5875477efa tamer: xir::Token: Remove span from Display
This was missed when removing it from other Display impls when the new
diagnostic system was introduced.  Raw `Span`s display byte offsets and the
context, which is no longer desirable as part of an error message.

DEV-10936
2022-05-03 09:09:55 -04:00
Mike Gerwitz 2ea66f4f97 tame: @encoding="{ISO-8859-1=>utf-8}" for all XML-based files
TAMER rejects this, because we shouldn't be using anything but UTF-8.  My
use of this encoding is ancient, from over a decade ago, that was apparently
just copied around.

DEV-10936
2022-05-02 12:00:42 -04:00
Mike Gerwitz a2e6e37ed1 tamer: Bump nightly Rust version 1.{57=>62}
This removes a couple of feature flags that are no longer necessary.
2022-05-02 11:05:32 -04:00
Mike Gerwitz 7248ef77e4 tamer: diagnose::resolve{r=>}: Rename
Consistent with naming of other modules, which prefers to not needlessly
transform words.

DEV-12151
2022-05-02 09:49:22 -04:00
Mike Gerwitz 75b966c577 tamer: diagnose: Additional documentation
I had waited to provide more documentation until I was sure that the
abstraction was not going to change significantly; there was a lot of
refactoring in prior commits.

DEV-12151
2022-05-02 09:44:53 -04:00
Mike Gerwitz fc1dad8483 tamer: diagnose::report::Section: Further refactor resolved constructor
This speaks for itself.

DEV-12151
2022-04-29 15:54:38 -04:00
Mike Gerwitz ba0ceddd2d tamer: diagnose::report::Section: Constructor refactoring
This moves construction out of `From` and into separate associated
functions, which can be further simplified in a bit.

We also need unit tests for this, since this still relies on integration
tests due to the cost of the aggressive and tight refactoring iterations.

DEV-12151
2022-04-29 13:10:04 -04:00
Mike Gerwitz 3e04217741 tamer: diagnose::report::Section::maybe_squash_into: Remove syslabel TODO
Previously, when adjacent duplicate spans were both resolved, if one failed,
the other certainly would, which would result in duplicate labels each
squash.  Elided spans do not have syslabels, and so this is no longer a
concern.

DEV-12151
2022-04-29 13:07:51 -04:00
Mike Gerwitz 2ae6df38e7 tamer: diagnose::report: Restore source line preview for invalid UTF-8
This was removed in a previous commit while working on simplifying the
implementation, with the hope of returning to it once things were in a
better place.  They are, so let's bring it back.

DEV-12151
2022-04-29 12:41:56 -04:00
Mike Gerwitz f8dda12fae tamer: diagnose::report: Remove TODOs that are no longer applicable
These relate to the most recent commits.

DEV-12151
2022-04-29 12:34:48 -04:00
Mike Gerwitz 2ce0dbdd84 tamer: diagnose::report::SpanLabel: Remove in favor of separate Level and Label
`SpanLabel` was created during a very early refactoring of this system, and
I've just been fighting with it sense.  This removes it, and simplifies
some things in the process.

It also makes clear that `Level` is never optional and removes the awkward
`Level::default` that was there previously; the default is now the lowest
level, which will always be able to be escalated.

DEV-12151
2022-04-29 12:13:11 -04:00
Mike Gerwitz 9a5a2c4f3f tamer: diagnose::report: Avoid re-resolving adjacent identical spans
This does what the original proof-of-concept implementation did---skip a
span that was just processed, since it'll be squashed into the previous
anyway.  These duplicate spans originate from the diagnostic system when
producing supplemental help information.

DEV-12151
2022-04-29 11:57:50 -04:00
Mike Gerwitz a533244473 tamer: diagnose::report::VisualReporter::render: Avoid mspan collection
This used to be necessary when `Report` stored references to heap-allocated
strings, but `Report` now owns those values itself.

DEV-12151
2022-04-29 09:53:22 -04:00
Mike Gerwitz b0a5265ad3 tamer: diagnose::report::test: Extract into separate file
Tests are large and will be getting larger.  The source will also grow as
it's better documented and cleaned up.  It's getting more difficult to
navigate efficiently and concurrently modify implementation and tests, and
parsing via LSP is getting slower with certain types of changes.

DEV-12151
2022-04-29 09:23:06 -04:00
Mike Gerwitz 5c0e224d3c tamer: diagnose::report: Line numbers in gutter
Alright, starting to settle on an abstraction now, and things are coming
together.  This gives us line numbers in the previously-empty gutter, and
widens the gutter to accommodate.  Gutters are normalized across
sections.  Sections are not yet collapsed for sequential line numbers in the
same context.

Exciting!

Here's an example, on an xmlo file:

error: expected closing tag for `preproc:symtable`
     --> /home/.../foo.xmlo:16:4
      |
   16 |    <preproc:symtable xmlns:map="http://www.w3.org/2005/xpath-functions/map">
      |    ----------------- note: element `preproc:symtable` is opened here

     --> /home/.../foo.xmlo:11326:4
      |
11326 |    </preproc:wrong>
      |    ^^^^^^^^^^^^^^^^ error: expected `</preproc:symtable>`

DEV-12151
2022-04-28 23:53:38 -04:00
Mike Gerwitz 5744e08984 tamer: diagnostic::report: Hoist gutter output into Section
The `Section` itself is now responsible for outputting the gutter, which
puts us in a position to be able to apply consistent formatting without
having to propagate width data to every line variant.
2022-04-28 22:59:13 -04:00
Mike Gerwitz 4e03a367a5 tamer: diagnose::report::SourceLine: Separate variants for each line
Now `SourceLine` _does_ actually correspond to a line of output, which will
allow for better formatting (e.g. collapsing padding) and, importantly,
proper management of gutters.

Note that the seemingly unnecessary `SectionSourceLine` allows for a subtle
consistent formatting for all variants' gutters in `SectionLine`, which will
allow us to hoist that rendering out in the next commit.  The other option
was to include a trailing space for padding and marks, but that is not only
sloppy and undesirable, but asking for confusion, especially in editors (like
mine) that trim trailing whitespace.

DEV-12151
2022-04-28 22:49:35 -04:00
Mike Gerwitz fd1c6430a8 tamer: diagnose::report::SectionSourceLine: {Option<Column>=>Column}
If a column isn't present, it degrades to displaying labels like footnotes
anyway, so this simplifies the system rather than catering to a rare
case.  With that said, this does lose functionality, since it does not
render the source line at all, even though we _could_ do so.

I may re-introduce that rendering after some further refactoring,
specifically for gutters.

DEV-12151
2022-04-28 22:23:58 -04:00
Mike Gerwitz 3a5dcfc016 tamer: diagnose::resolver::SourceLine: {Vec<u8>=>String}
Using a byte vector just makes life more difficult with regard to preparing
the diagnostic reports.  We're already validating UTF-8 data for column
generation, which is necessary for a robust report, so let's just store it
as a String to begin with.

DEV-12151
2022-04-28 22:03:37 -04:00
Mike Gerwitz 838db689ad tamer: diagnose::report: Render labels on mark line
Note that, if a span is first encountered with a mark but with _no_ label,
the first label (if collapsed) will be on the next line.  This allows a span
to be marked without extra visual noise if it's not necessary, and to be
able to trust that it'll stay that way.

Until coloring is introduced, this may or may not be easier to read
depending on context.

This is also not yet taking into account where on the line it begins, and so
may render poorly if the span is at the end of a line.  That will be fixed
later on.

DEV-12151
2022-04-28 16:23:13 -04:00
Mike Gerwitz a197267a2d tamer: xir::flat: Remove closing tag name from label
This is now visible in the diagnostic output.  Example at this point in
time, on an xmlo file for one of our smallest systems:

error: expected closing tag for `preproc:symtable`
  --> /home/.../foo.xmlo:16:4
   |
   |    <preproc:symtable xmlns:map="http://www.w3.org/2005/xpath-functions/map">
   |    -----------------
   = note: element `preproc:symtable` is opened here

  --> /home/.../foo.xmlo:11326:4
   |
   |    </preproc:wrong>
   |    ^^^^^^^^^^^^^^^^
   = error: expected `</preproc:symtable>`

DEV-12151
2022-04-28 15:47:34 -04:00
Mike Gerwitz 33baca113a tamer: diagnose::report: Vary mark character depending on level
Looking more and more Rust-like.  Shameless copy.

TBH I forget what character it uses for help, but it's easy enough to
change.

Also, to be clear: this is modeled after Rust, but it's not a requirement of
mine that it look exactly like it.  I just like the general style; I'll
surely deviate over time, as appropriate (or as I feel like it).

DEV-12151
2022-04-28 15:44:50 -04:00
Mike Gerwitz 8119d1ca0d tamer: diagnose::report: Render span marks under lines
This has the effect of highlighting the columns of the source lines using
'^' as an underline.

The next step will be to have the underline character depend on the
`Level`.

If this commit message doesn't sound all that exciting, given what it
finally achieved after all this time, it's because I'm exhausted, and my
prototype has already taken my excitement.  But this is significant, given
all the work leading up to it.

There is some code cleanup needed and some unit tests that ought to be
written rather than relying on integration, but considering how much this is
being refactored, I don't want to add to that refactoring cost just yet
before gutters are introduced and I know things are settled for now.

DEV-12151
2022-04-28 15:44:49 -04:00
Mike Gerwitz 5db026ed76 tamer: diagnose::report: Initial display of source lines
This has been a lot of refactoring for something that I prototyped a week
ago, and the prototype is still further along in its output formatting (it
has line numbering in gutters and span markings).

But, this has come a long way, and I'm happy with it overall, though I'm not
happy with my slow pace and struggle to maintain focus.  But those are
personal issues.

This leaves a lot to be desired, but at the same time is still really
helpful.  There's a couple notable TODOs regarding pointless allocation and
UTF8 re-checking, but otherwise, the feature-related steps are:

  - Gutters with line numbers; and
  - Marking columns associated with the span.

DEV-12151
2022-04-28 14:33:08 -04:00
Mike Gerwitz 3e06c9aaf3 tamer: diagnose::report: Prepare Section for output of source lines
This lowers the resolved span data into `Section` for display.  The next
step is to actually output it.

DEV-12151
2022-04-28 13:34:05 -04:00
Mike Gerwitz 331aada2bd tamer: diagnose::report::MaybeResolvedSpan: Move up in file
Just rearranging, since this was awkwardly placed relative to where it's
used.

DEV-12151
2022-04-28 11:00:36 -04:00
Mike Gerwitz 6a5a29c2f5 tamer: diagnose::report: Remove Section variants and eagerly squash
Rather than squashing as a separate operation, and explicitly denoting when
it occurred, we'll just always squash, as was done before these changes.  It
doesn't really make sense to make this optional and there's not any value in
keeping the decision around.

This also sets us up favorably for future changes: it creates a vector of
labels, which can be analyzed later to determine how to best lay out marks
and labels.

DEV-12151
2022-04-28 10:30:04 -04:00