Commit Graph

1216 Commits (838db689ad7787868ac55645352e1523544aebfc)

Author SHA1 Message Date
Mike Gerwitz 838db689ad tamer: diagnose::report: Render labels on mark line
Note that, if a span is first encountered with a mark but with _no_ label,
the first label (if collapsed) will be on the next line.  This allows a span
to be marked without extra visual noise if it's not necessary, and to be
able to trust that it'll stay that way.

Until coloring is introduced, this may or may not be easier to read
depending on context.

This is also not yet taking into account where on the line it begins, and so
may render poorly if the span is at the end of a line.  That will be fixed
later on.

DEV-12151
2022-04-28 16:23:13 -04:00
Mike Gerwitz a197267a2d tamer: xir::flat: Remove closing tag name from label
This is now visible in the diagnostic output.  Example at this point in
time, on an xmlo file for one of our smallest systems:

error: expected closing tag for `preproc:symtable`
  --> /home/.../foo.xmlo:16:4
   |
   |    <preproc:symtable xmlns:map="http://www.w3.org/2005/xpath-functions/map">
   |    -----------------
   = note: element `preproc:symtable` is opened here

  --> /home/.../foo.xmlo:11326:4
   |
   |    </preproc:wrong>
   |    ^^^^^^^^^^^^^^^^
   = error: expected `</preproc:symtable>`

DEV-12151
2022-04-28 15:47:34 -04:00
Mike Gerwitz 33baca113a tamer: diagnose::report: Vary mark character depending on level
Looking more and more Rust-like.  Shameless copy.

TBH I forget what character it uses for help, but it's easy enough to
change.

Also, to be clear: this is modeled after Rust, but it's not a requirement of
mine that it look exactly like it.  I just like the general style; I'll
surely deviate over time, as appropriate (or as I feel like it).

DEV-12151
2022-04-28 15:44:50 -04:00
Mike Gerwitz 8119d1ca0d tamer: diagnose::report: Render span marks under lines
This has the effect of highlighting the columns of the source lines using
'^' as an underline.

The next step will be to have the underline character depend on the
`Level`.

If this commit message doesn't sound all that exciting, given what it
finally achieved after all this time, it's because I'm exhausted, and my
prototype has already taken my excitement.  But this is significant, given
all the work leading up to it.

There is some code cleanup needed and some unit tests that ought to be
written rather than relying on integration, but considering how much this is
being refactored, I don't want to add to that refactoring cost just yet
before gutters are introduced and I know things are settled for now.

DEV-12151
2022-04-28 15:44:49 -04:00
Mike Gerwitz 5db026ed76 tamer: diagnose::report: Initial display of source lines
This has been a lot of refactoring for something that I prototyped a week
ago, and the prototype is still further along in its output formatting (it
has line numbering in gutters and span markings).

But, this has come a long way, and I'm happy with it overall, though I'm not
happy with my slow pace and struggle to maintain focus.  But those are
personal issues.

This leaves a lot to be desired, but at the same time is still really
helpful.  There's a couple notable TODOs regarding pointless allocation and
UTF8 re-checking, but otherwise, the feature-related steps are:

  - Gutters with line numbers; and
  - Marking columns associated with the span.

DEV-12151
2022-04-28 14:33:08 -04:00
Mike Gerwitz 3e06c9aaf3 tamer: diagnose::report: Prepare Section for output of source lines
This lowers the resolved span data into `Section` for display.  The next
step is to actually output it.

DEV-12151
2022-04-28 13:34:05 -04:00
Mike Gerwitz 331aada2bd tamer: diagnose::report::MaybeResolvedSpan: Move up in file
Just rearranging, since this was awkwardly placed relative to where it's
used.

DEV-12151
2022-04-28 11:00:36 -04:00
Mike Gerwitz 6a5a29c2f5 tamer: diagnose::report: Remove Section variants and eagerly squash
Rather than squashing as a separate operation, and explicitly denoting when
it occurred, we'll just always squash, as was done before these changes.  It
doesn't really make sense to make this optional and there's not any value in
keeping the decision around.

This also sets us up favorably for future changes: it creates a vector of
labels, which can be analyzed later to determine how to best lay out marks
and labels.

DEV-12151
2022-04-28 10:30:04 -04:00
Mike Gerwitz c8d919d0cc tamer: diagnose::report: {'l=>'d}
Just renames the lifetime to refer to the `Diagnostic`, rather than a
`Label` returned by it, which was all `'l` was previously used for.

Note that many labels have a `'static` lifetime; this doesn't change that or
somehow cause it to reallocate; the label must life _for at least `'d`_.

DEV-12151
2022-04-27 15:20:16 -04:00
Mike Gerwitz e2c68c5e84 tamer: diagnose::report: Avoid message copy
Rather than rendering the diagnostic `Display` message to a string only to
copy it to yet another buffer later on, this simply stores a reference to
the `Diagnostic` that was provided.  This also adds a type to the `Report`
associating it with the provided `Diagnostic`, which does seem appropriate,
given that the report was produced for it.

I should probably rename '{l=>d} now.

DEV-12151
2022-04-27 15:20:14 -04:00
Mike Gerwitz 3dbab881da tamer: diagnose::report: Produce Report object
Rather than writing to the provided `Write` object, this produces a `Report`
object.  While a lifetime still exists for the diagnostic data (labels,
specifically), I was able to remove the other lifetime resulting from
`ResolvedSpan` by transferring ownership of the data to the `Report`
itself.  Once actual source lines are integrated shortly, `Report` will
include those as well.

This has been a tedious process, but it's coming together.  Hopefully these
commits documenting the progressive and ugly refactoring are found useful by
some reader in the future.

DEV-12151
2022-04-27 15:00:30 -04:00
Mike Gerwitz 3679ff590c tamer: diagnose::report: Remove `L` type parameter
The line number was getting special treatment that is simply not worth the
cost (with regards to how burdensome it is on the type definitions).  This
simplifies things quite a bit.

If we want header customization in the future, we can worry about that in a
different way, or allow the header as a whole to be swapped out, rather than
its constituents.

DEV-12151
2022-04-27 14:23:58 -04:00
Mike Gerwitz 589f5e8c58 tamer: diagnose::report::HeadingLineNum: Compose HeadingColNum
`HeadingColNum` is no longer constructed by `HeadingLineNum`.  This both
narrows the types and required data (e.g. removing dummy values in test
cases), and reduces the coupling (by favoring composition, but still coupled
with the concrete type).

DEV-12151
2022-04-27 11:43:46 -04:00
Mike Gerwitz 7dbe25be05 tamer: diagnose::report::HeadingLineNum: Lower MaybeResolvedSpan
Same as the previous commit with `HeadingColNum`---this removes the
dependency on `MaybeResolvedSpan`.

DEV-12151
2022-04-27 11:28:17 -04:00
Mike Gerwitz 68f9f4d241 tamer: diagnose::report::HeadingColNum: Lower MaybeResolvedSpan
This eliminates `MaybeResolvedSpan` from `HeadingColNum`, along with its
type parameters and lifetimes.

DEV-121251
2022-04-27 11:10:16 -04:00
Mike Gerwitz f29918b5a0 tamer: diagnose::report: Continue refactoring into report components
I'm unhappy with the current state of this, which is why I haven't settled
on docs or unit tests for these changes yet (though note that the
integration tests do cover these changes)---this is still a prototype
refactoring.

In particular, this needs to do more lowering---the `ResolvedSpan` and
`MaybeResolvedSpan` need to be eliminated and lowered into exactly what is
needed so that we can stop reasoning about them and propagating them.

Further, having lines and columns lazily evaluate themselves for
display---based on `MaybeResolvedSpan`---adds extra generics that shouldn't
be necessary; they should be pre-computed and store the concrete data they
need in variants.  Display shouldn't involve computation beyond formatting
of pre-computed data.

That was always the plan, but this refactoring has been incremental.

Anyway: this is in a working and integration-tested state, but it's going to
change.

DEV-12151
2022-04-27 10:48:41 -04:00
Mike Gerwitz e2f9d71c1f tamer: diagnose::report: Refined report components
This generalizes the types a bit more and introduces unit tests.  Note that
these are still also covered by integration tests.

The next step will be to finish generalizing
`<VisualReporter as Reporter>::render`, after which I'll get back to the
task of outputting the source line along with markings and labels.

DEV-12151
2022-04-26 13:26:52 -04:00
Mike Gerwitz d05bcaab03 tamer: {Resolved,Span}::{ctx=>context}: Rename
This is just to provide clarity.  `ctx` is not so widely used that we
benefit from such a short identifier, and it's not worth the cognitive
burden of people unfamiliar with what it may mean.

DEV-12151
2022-04-26 10:52:32 -04:00
Mike Gerwitz 16d76b95d0 tamer: diagnose::resolver::ResolvedSpanData: New trait
This provides the methods originally implemented on `ResolvedSpan` itself,
which will allow for mocking for unit testing.

DEV-12151
2022-04-26 10:46:47 -04:00
Mike Gerwitz 0928427116 tamer: diagnose::resolver::Column::At: Remove
This is redundant with the `Endpoints` variant, although it did read
better.  It's just another case to have to handle.

I was originally going to use `std::ops::RangeInclusive` for `Endpoints`,
however that struct also contains an extra bool indicating whether it was
exhausted (as an iterator), which isn't appropriate for this.

DEV-12151
2022-04-26 10:30:07 -04:00
Mike Gerwitz ec93488365 tamer: diagnost::resolver::ResolvedSpan: Clear methods for all data
This (a) makes it clear the intent of these methods and (b) will allow
introducing a trait for mocking it.

DEV-12151
2022-04-26 10:22:31 -04:00
Mike Gerwitz b9ff7770aa tamer: diagnose::report: Begin refactoring into Display impls
This logic is still covered by the integration tests; I'll be adding unit
tests once it's decoupled to the point where that's possible, which should
be shortly, and after I make sure this is the route I do want to go down.

DEV-12151
2022-04-26 10:14:51 -04:00
Mike Gerwitz c0ace258f0 tamer: diagnose::resolver::SourceLine:: Guarantee non-empty lines
This simplifies types and error handling since we will always have at least
one line, provided that the span is within the range of the context.  To
ensure that, this patch introduces a new error.

DEV-12151
2022-04-22 16:50:16 -04:00
Mike Gerwitz 56b8aec9b7 tamer: diagnose::resolver::test: Extract into own file
There's just a lot here.

DEV-12151
2022-04-22 15:31:12 -04:00
Mike Gerwitz 2e0925627e tamer: diagnose::Label: Introduce lifetime and inner Cow
I did not initially introduce lifetimes because I wasn't sure how the system
was going to evolve, but now lifetimes are going to be needed in a number of
contexts.  The core of TAMER is able to avoid lifetimes in most instances
because of its internment system, but its use is not appropriate for the
diagnostic system's buffers (beyond sourcing strings from already-interned
data).

DEV-12151
2022-04-22 13:23:53 -04:00
Mike Gerwitz aeff7aeed3 tamer: diagnose::test: Extract into own file
This is going to get quite large over time.

DEV-12151
2022-04-22 09:21:18 -04:00
Mike Gerwitz 596c9de85e tamer: diagnose::resolver::SourceLine (line=>num): Rename
`line.line` was rather confounding.

DEV-12151
2022-04-21 15:47:15 -04:00
Mike Gerwitz 5b1f0ab6c6 tamer: diagnostic: Column resolution
Determining the column number is not as simple as performing byte
arithmetic, because certain characters have different widths.  Even if we
only accepted ASCII, control characters aren't visible to the user.

This uses the unicode-width crate as an alternative to POSIX wcwidth, to
determine (hopefully) the number of fixed-width cells that a unicode
character will take up on a terminal.  For example, control characters are
zero-width, while an emoji is likely double-width.  See test cases for more
information on that.

There is also the unicode-segmentation crate, which can handle extended
grapheme clusters and such, but (a) we'll be outputting the line to the
terminal and (b) there's no guarantee that the user's editor displays
grapheme clusters as a single column.  LSP measures in UTF-16,
apparently.  I use both Emacs and Vim from a terminal, so unicode-width
applies to me.  There's too much variation to try to solve that right now.

The columns can be considered a visual span---this gives us enough
information to draw line annotations, which will happen soon.

Here are some useful links:

  - https://hsivonen.fi/string-length/
  - https://unicode.org/reports/tr29/
  - https://github.com/rust-analyzer/rowan/issues/17
  - https://www.reddit.com/r/rust/comments/gpw2ra/how_is_the_rust_compiler_able_to_tell_the_visible/

DEV-10935
2022-04-21 14:27:36 -04:00
Mike Gerwitz e555955450 tamer: span::Span::endpoints_saturated: New method
This gets rid of the `Option` and is used in the diagnostic system (next
commit).

DEV-10935
2022-04-21 14:15:25 -04:00
Mike Gerwitz a22e8e79f7 tamer: diagnose: Integrate resolver for source lines
This does not yet resolve columns, and omits the length of the span, but
it's starting to come together.

This is particularly exciting for me to see because I've been wanting line
numbers in TAME error messages for over a decade.

DEV-10935
2022-04-21 12:34:17 -04:00
Mike Gerwitz 9b4c84de26 tamer: diagnose::resolver: Support rewinding
This does adds support for rewinding the underlying buffer when necessary to
read a span that occurs earlier within the same context (which could also
include the same span read twice).

As part of this change, I cleaned up the code a bit.  Working with this
system can be confusing with the different meanings of the byte offsets and
the different ways of interpreting lines relative to the span that is
provided.  There's not a lot of code here, but it represents a lot of work
to get right.
2022-04-21 12:33:27 -04:00
Mike Gerwitz 1b02e77537 tamer: span (SpanOffsetSize, SpanLenSize): New type aliases
Callers can use these types instead of having to reference globals.

DEV-10935
2022-04-20 09:42:13 -04:00
Mike Gerwitz ab48d79e1f tamer: diagnost::resolver: Initial concept for line resolution
This works, but it's ugly and requires some cleanup.  It shows that there
are some interesting considerations when determining how to best represent
the location of spans to the user in a way that is intuitive.

This is not yet integrated with the reporter, which will require a layer to
load a `Context` from disk.

DEV-10935
2022-04-20 09:42:13 -04:00
Mike Gerwitz a77eb7d937 tamer: span: Minor test refactoring
Just some cleanup based on some new conventions, now that I'm about to make
some changes.

DEV-10935
2022-04-20 09:42:12 -04:00
Mike Gerwitz 725dc3fb54 tamer: tamec: Use diagnostic system for errors
This is a POC, minimal-effort integration that also creates the TamecError
sum type analogous to TameldError.

I'll work on reducing the boilerplate in the future.

A note regarding the type and boilerplate vs. dynamic dispatch, for any
future readers: the purpose of this is to be explicit about the error types
so that the system is self-documenting and it forces and understanding of
its error conditions.  `Box<dyn Error>` is basically "eh idk anything can
happen!", which is not what I'm interested in having.

DEV-10935
2022-04-20 09:42:11 -04:00
Mike Gerwitz eaa8133d21 tamer: diagnose: Introduction of diagnostic system
This is a working concept that will continue to evolve.  I wanted to start
with some basic output before getting too carried away, since there's a lot
of potential here.

This is heavily influenced by Rust's helpful diagnostic messages, but will
take some time to realize a lot of the things that Rust does.  The next step
will be to resolve line and column numbers, and then possibly include
snippets and underline spans, placing the labels alongside them.  I need to
balance this work with everything else I have going on.

This is a large commit, but it converts the existing Error Display impls
into Diagnostic.  This separation is a bit verbose, so I'll see how this
ends up evolving.

Diagnostics are tied to Error at the moment, but I imagine in the future
that any object would be able to describe itself, error or not, which would
be useful in the future both for the Summary Page and for query
functionality, to help developers understand the systems they are writing
using TAME.

Output is integrated into tameld only in this commit; I'll add tamec
next.  Examples of what this outputs are available in the test cases in this
commit.

DEV-10935
2022-04-13 15:22:46 -04:00
Mike Gerwitz 702b5ebb23 tamer: span: Remove PathIndex
We can just use PathSymbolId directly and simplify things.  Typing can (and
should) happen on the symbol itself, and if we want a separate symbol type,
it ought to have its own interner.

For now, it doesn't, and having this extra type is just a PITA.

DEV-10935
2022-04-13 09:59:11 -04:00
Mike Gerwitz c49510646b tamer: parse::Parser (last_span): Replace Option with UNKNOWN_SPAN
There's no use in complicating the error handling here when we'd just
default to `UNKNOWN_SPAN` anyway when trying to render it.  `UNKNOWN_SPAN`
didn't exist at the time of writing.

DEV-10935
2022-04-12 09:59:00 -04:00
Mike Gerwitz cfc7f45bc4 tamer: Remove wip-xmlo-xir-reader
This entirely removes the old XmloReader that has since been replaced with a
XIR-based reader.

I had been holding off on this because the new reader is slower, pending
performance optimizations (which I'll do a little later on), however the
performance loss is of no practical consideration and only affects the
linker, which is still fast.

Therefore, it's better to get this old code out of the way to simplify
refactoring going forward.  In particular, I'm working on the diagnostic
system.

This is a little sad, in a way---this is some of my first Rust code that I'm
deleting.

DEV-10935
2022-04-11 16:11:49 -04:00
Mike Gerwitz 4c69efd175 tamer: obj::xmlo::error: Remove XirfError
This does not deal directly with XIRF (that's composed into a pipeline
outside of this parser).

I'd like to clean up further...perhaps I should retire the
wip-xmlo-xir-reader flag now, despite the minor performance regression (see
previous recent commits for explanation).

DEV-10935
2022-04-11 15:52:40 -04:00
Mike Gerwitz f07c0e75be tamer: tameld (TameldError): Error sum type
This aggregates all non-panic errors that can occur during link time, making
`Box<dyn Error>` unnecessary.  I've been wanting to do this for a long time,
so it's nice seeing this come together.  This is a powerful tool, in that we
know, at compile time, all errors that can occur, and properly report on
them and compose them.  This method of error composition ensures that all
errors have a chance to be handled within their context, though it'll take
time to do so in a decent way.

This just maintains compatibility with the dynamic dispatch that was
previous occurring.  This work is being done to introduce the initial
diagnostic system, which was really difficult/confusing to do without proper
errors types at the top level, considering the toplevel is responsible for
triggering the diagnostic reporting.

The cycle error is in particular going to be interesting once the system is
in place, especially once it provides spans in the future, since it will
guide the user through the code to understand how the cycle formed.

More to come.

DEV-10935
2022-04-11 15:15:04 -04:00
Mike Gerwitz a1a4ad3e8e tamer: Introduce context into XirReader
tamec and tameld will now both introduce a `Context` to XIR, which will use
it to create spans.

Here's an example of an error, now that it's all working well together:

  $ target/release/tameld --emit xmle -o /dev/null path/to/package.xmlo
  error: invalid preproc:sym/@dim `9` at [/../path/to/package.xmlo offset 1175451-1175452]

A future task will make this human-readable by producing line and column
numbers, and perhaps even a snippet (if not now, then eventually).

It's exciting to see this coming together finally.

DEV-10934
2022-04-08 16:16:23 -04:00
Mike Gerwitz 68223cb7d3 tamer: xir::reader: Additional quick-xml error spans
There's a bit to unpack here.  Some of the spans originate from quick-xml's
error handling, but in coming up with test cases to try to trigger errors, I
found that quick-xml is far too permissive in what it accepts, and
oughtright dangerous in some situations.

I feel like the writing is on the wall for quick-xml, but I'll probably wait
until replacing `xmlo` with a more efficient format before deciding whether
to use a different library or implement parsing ourselves.  There's a lot of
factors to consider, and a library would have to not only be correct and
performant, but provide useful information for span generation.

But for now, I have other more important things to work on, like a
functioning compiler.  So while quick-xml is around, I'll just have to do
the best I can to provide a correct parser with useful errors.

DEV-10934
2022-04-08 14:54:49 -04:00
Mike Gerwitz ab181670b5 tamer: xir::reader: Initial introduction of spans
This is a large change, and was a bit of a tedious one, given the
comprehensive tests.

This introduces proper offsets and lengths for spans, with the exception of
some quick-xml errors that still need proper mapping.  Further, this still
uses `UNKNOWN_CONTEXT`, which will be resolved shortly.

This also introduces `SpanlessError`, which `Error` explicitly _does not_
implement `From<SpanlessError>` for---this forces the caller to provide a
span before the error is compatable with the return value, ensuring that
spans will actually be available rather than forgotten for errors.  This is
important, given that errors are generally less tested than the happy path,
and errors are when users need us the most (so, need span information).

Further, I had to use pointer arithmetic in order to calculate many of the
spans, because quick-xml does not provide enough information.  There's no
safety considerations here, and the comprehensive unit test will ensure
correct behavior if the implementation changes in the future.

I would like to introduce typed spans at some point---I made some
opinionated choices when it comes to what the spans ought to
represent.  Specifically, whether to include the `<` or `>` with the open
span (depends), whether to include quotes with attribute values (no),
and some other details highlighted in the test cases.  If we provide typed
spans, then we could, knowing the type of span, calculate other spans on
request, e.g. to include or omit quotes for attributes.  Different such
spans may be useful in different situations when presenting information to
the user.

This also highlights gaps in the tokens emitted by XIR, such as whitespace
between attributes, the `=` between name and value, and so on.  These are
important when it comes to code formatting, so that we can reliably
reconstruct the XML tree, but it's not important right now.  I anticipate
future changes would allow the XIR reader to be configured (perhaps via
generics, like a strategy-type pattern) to optionally omit these tokens if
desired.

Anyway, more to come.

DEV-10934
2022-04-08 13:59:37 -04:00
Mike Gerwitz 942bf66231 tamer: frontend: Clean up unused modules
These were part of a POC for frontends quite some time ago.  Some portions
of this concept may be reintroduced, but this was pre-XIR.

DEV-10413
2022-04-07 14:21:08 -04:00
Mike Gerwitz 99aacaf7ca tamer: tamec: Replace copy with XIR parsing/writing
When wip-frontends is on, this will parse the input file using XIR and then
immediately output it again.  This makes the necessary changes to be able to
read every source file we have in our largest project, such that the output
is identical after having been formatted with `xmllint --format -` (there
are differences because e.g. whitespace between attributes is not yet
maintained).

This is performant too, with times remaining essentially identical despite
the additional work.

DEV-10413
2022-04-07 12:13:49 -04:00
Mike Gerwitz b90bf9d8a8 tame: build-aux/{csv2xml,tdat2xml}: Remove xml-stylesheet XML PI
These declarations are relics from when all XML files could be loaded in the
browser to render the Summary Page.  Such a thing has not worked for many
years.

The previous commit will cause files produced by these scripts to be
regenerated.

I noticed this when reading source files using XIR.

DEV-10413
2022-04-07 09:32:00 -04:00
Mike Gerwitz 8e9b2a7211 tame: build-aux/Makefile.am: Generated sources depend on scripts that generate them
This ensures that, when changes are made to these scripts, the files that
are generated from them are re-generated.

Historically this probably was not noticed because (a) they seldom changed
and (b) we had a small team and I told people to re-run bootstrapping
scripts or clean files.  The team is much larger now, and regardless,
there's no reason not to have had this in place.

DEV-10413
2022-04-07 09:31:55 -04:00
Mike Gerwitz 2e386f1baf tamer: xir::reader::XmlXirReader::refill_buf: Clear read buffer
This was done in the old reader many months ago, but I somehow forgot to do
it here (or forgot to).  The new reader was using substantially more memory.

Here's how this change affects the memory profile for one of our
systems (output from `ms_print`):

Before:

    MB
79.75^                                                             #
     |                                                             #
     |                                                             #       @
     |                                               @@@@          #       @
     |                                               @@@           #      @@
     |                                               @@@        @@@#@   @@@@@
     |                                               @@@        @@ #@@@@@@@@@@
     |                                            @@@@@@      @@@@ #@@@@@@@@@@
     |                                         @@ @@ @@@   @@ @ @@ #@@@@@@@@@@
     |                                         @@ @@ @@@  @@@@@ @@ #@@@@@@@@@@
     |                                         @@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     |                                         @@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     |   @@                                    @@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     |   @        @@     @@          @        @@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     |   @        @     @@@         @@  @@@   @@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     |   @     @@@@ @@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     | @@@   @@@@@@ @@@@@@@@@ @@@@@ @@@@@ @@ @@@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     | @@@   @ @@@@ @@@@@@@@@ @@@@@ @@@@@ @@ @@@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     | @@@ @@@ @@@@ @@@@@@@@@ @@@@@ @@@@@ @@ @@@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
     | @@@ @@@ @@@@ @@@@@@@@@ @@@@@ @@@@@ @@ @@@@@@@ @@@ @@@@@@ @@ #@@@@@@@@@@
   0 +----------------------------------------------------------------------->Gi
     0                                                                   15.20

After:

    MB
63.25^                                                                      #
     |                                                                      #
     |                                                             @@@@@@@@@#@
     |                                                             @@@@@@ @@#@
     |                                                             @@@@@@ @@#@
     |                                                             @@@@@@ @@#@
     |                                                             @@@@@@ @@#@
     |                                                       @@@@@@@@@@@@ @@#@
     |                                                @@@@@@@@@ @@ @@@@@@ @@#@
     |                                         @@@@@@@@ @@@ @@@ @@ @@@@@@ @@#@
     |                                         @@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |                                         @@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |                                        @@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |                                        @@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |           @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |        @@@@@@@@@@@@@ @@@@@@@@ @@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |      @@@@@@@@@@@@@@@ @@@@@@@@ @@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     |    @@@@@@@@@@@@@@@@@ @@@@@@@@ @@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     | @@@@@@@@@@@@@@@@@@@@ @@@@@@@@ @@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
     | @@@@@@@@@@@@@@@@@@@@ @@@@@@@@ @@@@@@@@@@@@@@@  @ @@@ @@@ @@ @@@@@@ @@#@
   0 +----------------------------------------------------------------------->Gi
     0                                                                   15.20

The bottom graph is virtually identical to the memory profile of the old
reader, just with the exception that it's interning a bit more data than
before, because we're reading more comprehensively.

That's (potentially) the subject of future changes.

DEV-12038
2022-04-06 11:50:07 -04:00
Mike Gerwitz 6871a0cdc7 tamer: parse (ParseState): Doc correction regarding determinism
The pair is now a triple and parsers are often NFAs.
2022-04-05 15:55:58 -04:00