Commit Graph

115 Commits (26aaf6efc159002fe028871dfb82f19b5240e075)

Author SHA1 Message Date
Mike Gerwitz 1c181fe546 tamer: parse::lower: Propagate widened errors to terminal parser
The term "terminal parser" isn't formalized yet in the system, but is meant
to refer to the innermost parser that is responsible for pulling tokens
through the lowering pipeline.

This approach is more of what one would expect when dealing with
`Result`-like monads---we are effectively chaining the inner operation while
propagating errors to short-circuit lowering and let the caller decide
whether recovery ought to be permitted with diagnostic messages.  This will
become more clear as it is further refactored.

This also means that the previous changes for introducing interior
mutability for a shared mutable `Reporter` can be reverted, which is great,
since that approach was antithetical to how the streaming pipeline
operates (and introduces awkward mutable state into an
otherwise-mostly-immutable system).

DEV-13158
2022-10-26 12:32:51 -04:00
Mike Gerwitz 65b42022f0 tamer: xir::st: Prefix all preproc-namespaced constants with `QN_P_`
I had previously avoided this to keep names more concise, but now it's
ambiguous with parsing actual TAME sources.

DEV-7145
2022-08-15 13:00:10 -04:00
Mike Gerwitz 7a5f731cac tamer: tameld: XIRF nesting 64=>4
Since we'll never be reading past the header, this is all that is needed.

If in the future this is violated, XIRF will cause a nice diagnostic error
displaying precisely what opening tag caused the increased level of nesting,
which will aid in debugging and allow us to determine if it ought to be
increased.  Here's an example, if I set the max to `3`:

  error: maximum XML element nesting depth of `3` exceeded
     --> /home/.../foo.xmlo:261:10
      |
  261 |          <preproc:sym-ref name=":_vproduct:vector_a"/>
      |          ^^^^^^^^^^^^^^^^ error: this opening tag increases the level of nesting past the limit of 3

Of course, the longer-term goal is to do away with `xmlo` entirely.

This had no (perceivable via `/usr/bin/time -v`, at least) impact on memory
or CPU time.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz 41b41e02c1 tamer: Xirf::Text refinement
This teaches XIRF to optionally refine Text into RefinedText, which
determines whether the given SymbolId represents entirely whitespace.

This is something I've been putting off for some time, but now that I'm
parsing source language for NIR, it is necessary, in that we can only permit
whitespace Text nodes in certain contexts.

The idea is to capture the most common whitespace as preinterned
symbols.  Note that this heuristic ought to be determined from scanning a
codebase, which I haven't done yet; this is just an initial list.

The fallback is to look up the string associated with the SymbolId and
perform a linear scan, aborting on the first non-whitespace character.  This
combination of checks should be sufficiently performant for now considering
that this is only being run on source files, which really are not all that
large.  (They become large when template-expanded.)  I'll optimize further
if I notice it show up during profiling.

This also frees XIR itself from being concerned by Whitespace.  Initially I
had used quick-xml's whitespace trimming, but it messed up my span
calculations, and those were a pain in the ass to implement to begin with,
since I had to resort to pointer arithmetic.  I'd rather avoid tweaking it.

tameld will not check for whitespace, since it's not important---xmlo files,
if malformed, are the fault of the compiler; we can ignore text nodes except
in the context of code fragments, where they are never whitespace (unless
that's also a compiler bug).

Onward and yonward.

DEV-7145
2022-08-01 15:01:37 -04:00
Mike Gerwitz c671bf6a9c tamer: xir: Introduce {Ele,Open,Close}Span
This isn't conceptally all that significant of a change, but there was a lot
of modify to get it working.  I would generally separate this into a commit
for the implementation and another commit for the integration, but I decided
to keep things together.

This serves a role similar to AttrSpan---this allows deriving a span
representing the element name from a span representing the entire XIR
token.  This will provide more useful context for errors---including the tag
delimiter(s) means that we care about the fact that an element is in that
position (as opposed to some other type of node) within the context of an
error.  However, if we are expecting an element but take issue with the
element name itself, we want to place emphasis on that instead.

This also starts to consider the issue of span contexts---a blob of detached
data that is `Span` is useful for error context, but it's not useful for
manipulation or deriving additional information.  For that, we need to
encode additional context, and this is an attempt at that.

I am interested in the concept of providing Spans that are guaranteed to
actually make sense---that are instantiated and manipulated with APIs that
ensure consistency.  But such a thing buys us very little, practically
speaking, over what I have now for TAMER, and so I don't expect to actually
implement that for this project; I'll leave that for a personal
project.  TAMER's already take a lot of my personal interests and it can
cause me a lot of grief sometimes (with regards to letting my aspirations
cause me more work).

DEV-7145
2022-06-24 14:16:29 -04:00
Mike Gerwitz 2b8e7e6031 tamer: xir::st::qname: New module
This moves and deduplicates the static `QName`s into a common area.

DEV-7145
2022-06-06 11:31:27 -04:00
Mike Gerwitz 3da82b351e tamer: xir::flat::{State=>XirToXirf}: Rename
Like the previous two commits, this states the intent of this parser, which
results in more clear pipeline composition.

DEV-7145
2022-06-02 13:48:54 -04:00
Mike Gerwitz 91b55999e2 tamer: asg::air::{AirState=>AirAggregate}: Rename
Like the previous commit, this emphasizes what is happening.

DEV-7145
2022-06-02 13:26:46 -04:00
Mike Gerwitz 45bbf3879e tamer: obj::xmlo::{lower=>air}: Rename {LowerState=>XmloToAir}
This provides much more clarity as to what is going on.  Further, it's less
ambiguous, since I'm about to introduce a new type of xmlo lowering into XIR
for writing the actual xmlo files.

DEV-7145
2022-06-02 13:23:41 -04:00
Mike Gerwitz 8d92667388 tamer: Integrate xir::reader as a parser in the lowering pipeline
This allows `XmlXirReader` to be used in a `Lower` operation, just as
everything else, bringing me one step closer to a pipeline that can be
concisely represented; this is finally beginning to unify in a clear way,
though it is still a bit of a mess.

This causes `XmlXirReader` to _act_ like a `parse::Parser` in that it yields
a `ParsedResult`, but it does not use `parse::Parser` itself; that was the
_original_ plan: convert it into a `ParseState` where `XmlXirReader` became
a context, and force `Parser` to yield by feeding it a stream of tokens with
`repeat`, but that ended up performing poorly relative to this change.  I
did some investigation, which I might write about in the future, but for
now, this solution works just fine.

DEV-7145
2022-06-02 10:30:44 -04:00
Mike Gerwitz 63aa452197 tamer: parse: Move parse::lower into Lower
This also modifies `poc` such that `Lower` is invoked as an associated
function rather than a method to emphasize the pattern that is forming, so
that it can be later abstracted away.

DEV-11864
2022-06-01 11:15:43 -04:00
Mike Gerwitz f40f8bbafc tamer: parse: Rename {lower_*_while_ok=>lower_*}
The `while_ok` can just be implied with a lowering operation, and that
reduces the name complexity so that we can maybe introduce even more
specialized methods without resulting in a huge sentence as a name.

DEV-11864
2022-05-27 14:10:55 -04:00
Mike Gerwitz b084e23497 tamer: Refactor asg_builder into obj::xmlo::lower and asg::air
This finally uses `parse` all the way up to aggregation into the ASG, as can
be seen by the mess in `poc`.  This will be further simplified---I just need
to get this committed so that I can mentally get it off my plate.  I've been
separating this commit into smaller commits, but there's a point where it's
just not worth the effort anymore.  I don't like making large changes such
as this one.

There is still work to do here.  First, it's worth re-mentioning that
`poc` means "proof-of-concept", and represents things that still need a
proper home/abstraction.

Secondly, `poc` is retrieving the context of two parsers---`LowerContext`
and `Asg`.  The latter is desirable, since it's the final aggregation point,
but the former needs to be eliminated; in particular, packages need to be
worked into the ASG so that `found` can be removed.

Recursively loading `xmlo` files still happens in `poc`, but the compiler
will need this as well.  Once packages are on the ASG, along with their
state, that responsibility can be generalized as well.

That will then simplify lowering even further, to the point where hopefully
everything has the same shape (once final aggregation has an abstraction),
after which we can then create a final abstraction to concisely stitch
everything together.  Right now, Rust isn't able to infer `S` for
`Lower<S, LS>`, which is unfortunate, but we'll be able to help it along
with a more explicit abstraction.

DEV-11864
2022-05-27 13:51:29 -04:00
Mike Gerwitz f218c452b9 tamer: iter::trip: Flatten Result
The `*_iter_while_ok` functions now compose like monads, flattening `Result`
at each step and drastically simplifying handling of error types.  This also
removes the bunch of `?`s at the end of the expression, and allows me to use
`?` within the callback itself.

I had originally not used `Result` as the return type of the callback
because I was not entirely sure how I was going to use them, but it's now
clear that I _always_ use `Result` as the return type, and so there's no use
in trying to be too accommodating; it can always change in the future.

This is desirable not just for cleanup, but because trying to refactor
`asg_builder` into a pair of `Parser`s is really messy to chain without
flattening, especially given some state that has to leak temporarily to the
caller.  More on that in a future commit.

DEV-11864
2022-05-20 16:08:16 -04:00
Mike Gerwitz 958a707e02 tamer: asg: Hoist Root from Ident into Object
This was always the intent, but I didn't have a higher-level object
yet.  This removes all the awkwardness that existed with working the root
in as an identifier.

DEV-11864
2022-05-19 12:48:43 -04:00
Mike Gerwitz 6252758730 tamer: asg::Object: Introduce Object::Ident
This wraps `Ident` in a new `Object` variant and modifies `Asg` so that its
nodes are of type `Object`.

This unfortunately requires runtime type checking.  Whether or not that's
worth alleviating in the future depends on a lot of different things, since
it'll require my own graph implementation, and I have to focus on other
things right now.  Maybe it'll be worth it in the future.

Note that this also gets rid of some doc examples that simply aren't worth
maintaining as the API evolves.

DEV-11864
2022-05-19 12:33:59 -04:00
Mike Gerwitz ebf1de5a60 tamer: asg::Ident{Object=>}: Rename
I think this may have been renamed _from_ `Ident` some time ago, but I'm too
lazy to check.  In any case, the name is redundant.

DEV-11864
2022-05-19 11:17:04 -04:00
Mike Gerwitz 07d2ec1ffb tamer: Move Dim and {Sym=>}Dtype into num module
A previous commit mentioned that there's not a place for `Dim`, and
duplicated it between `asg` and `xmlo`.  Well, `Dtype` is also needed in
both, and so here's a home for now.

`Dtype` has always been an inappropriate detail for the system and will one
day be removed entirely in favor of higher-level types; the machine
representation is up to the compiler to decide.

DEV-11864
2022-05-19 10:39:21 -04:00
Mike Gerwitz 8948452b71 tamer: asg::ident::Dim: Narrow type
This matches xmlo::Dim, and could be the same thing, if we can find a home
for it in the future; it's not worth creating such a home right now when I'm
not yet sure what else ought to live there; the duplication may be fine.

The conversion from xmlo needs to be moved, and `Dim` is going to be used
for more than just identifiers (expressions will have type inference
performed).

DEV-11864
2022-05-19 09:32:43 -04:00
Mike Gerwitz 3e277270a7 tamer: asg: Track roots on graph
Previously, since the graph contained only identifiers, discovered roots
were stored in a separate vector and exposed to the caller.  This not only
leaked details, but added complexity; this was left over from the
refactoring of the proof-of-concept linker some time ago.

This moves the root management into the ASG itself, mostly, with one item
being left over for now in the asg_builder (eligibility classifications).

There are two roots that were added automatically:

  - __yield
  - __worksheet

The former has been removed and is now expected to be explicitly mapped in
the return map, which is now enforced with an extern in `core/base`.  This
is still special, in the sense that it is explicitly referenced by the
generated code, but there's nothing inherently special about it and I'll
continue to generalize it into oblivion in the future, such that the final
yield is just a convention.

`__worksheet` is the only symbol of type `IdentKind::Worksheet`, and so that
was generalized just as the meta and map entries were.

The goal in the future will be to have this more under the control of the
source language, and to consolodate individual roots under packages, so that
the _actual_ roots are few.

As far as the actual ASG goes: this introduces a single root node that is
used as the sole reference for reachability analysis and topological
sorting.  The edges of that root node replace the vector that was removed.

DEV-11864
2022-05-17 10:42:05 -04:00
Mike Gerwitz 34eb994a0d tamer: asg::Asg::set_fragment: {ObjectRef=>SymbolId}
In the actual implementation (outside of tests), this is always looking up
before adding the symbol.  This will simplify the API, while still retaining
errors, since the identifier will fail the state transition if the
identifier did not exist before attempting to set a fragment.  So while this
is slower in microbenchmarks, this has no effect on real-world performance.

Further, I'm refactoring toward a streaming ASG aggregation, which is a lot
easier if we do not need to perform lookups in a separate step from the
ASG's primitives.

DEV-11864
2022-05-16 13:14:27 -04:00
Mike Gerwitz d87006391e tamer: asg::object: Remove IdentObjectState, IdentObjectData
These traits are no longer necessary now that I'm using concrete types; they
just add unnecessary noise and confusion as I attempt to further refactor.

Don't abstract prematurely.

DEV-11864
2022-05-12 16:31:36 -04:00
Mike Gerwitz 3748762d31 tamer: asg::graph::Asg: Remove type parameter O
This removes the generic on the Asg (which was formerly BaseAsg),
hard-coding `IdentObject`, which will further evolve.  This makes the IR an
actual concrete IR rather than an abstract data structure.

These tests bring me back a bit, since they were written as I was still
becoming familiar with Rust.

DEV-11864
2022-05-12 15:46:17 -04:00
Mike Gerwitz f2c5443176 tamer: asg: Remove generic Asg, rename {Base=>}Asg
This is the beginning of an incremental refactoring to remove generics, to
simplify the ASG.  When I initially wrote the linker, I wasn't sure what
direction I was going in, but I was also negatively influenced by more
traditional approaches to both design and unit testing.

If we're going to call the ASG an IR, then it needs to be one---if the core
of the IR is generic, then it's more like an abstract data structure than
anything.  We can abstract around the IR to slice it up into components that
are a little easier to reason about and understand how responsibilities are
segregated.

DEV-11864
2022-05-11 16:47:13 -04:00
Mike Gerwitz 1ad2fb1dc8 Copyright year update 2022
RSG (Ryan Specialty Group) recently announced a rename to Ryan Specialty (no
"Group"), but I'm not sure if the legal name has been changed yet or not, so
I'll wait on that.
2022-05-03 14:14:29 -04:00
Mike Gerwitz eaa8133d21 tamer: diagnose: Introduction of diagnostic system
This is a working concept that will continue to evolve.  I wanted to start
with some basic output before getting too carried away, since there's a lot
of potential here.

This is heavily influenced by Rust's helpful diagnostic messages, but will
take some time to realize a lot of the things that Rust does.  The next step
will be to resolve line and column numbers, and then possibly include
snippets and underline spans, placing the labels alongside them.  I need to
balance this work with everything else I have going on.

This is a large commit, but it converts the existing Error Display impls
into Diagnostic.  This separation is a bit verbose, so I'll see how this
ends up evolving.

Diagnostics are tied to Error at the moment, but I imagine in the future
that any object would be able to describe itself, error or not, which would
be useful in the future both for the Summary Page and for query
functionality, to help developers understand the systems they are writing
using TAME.

Output is integrated into tameld only in this commit; I'll add tamec
next.  Examples of what this outputs are available in the test cases in this
commit.

DEV-10935
2022-04-13 15:22:46 -04:00
Mike Gerwitz cfc7f45bc4 tamer: Remove wip-xmlo-xir-reader
This entirely removes the old XmloReader that has since been replaced with a
XIR-based reader.

I had been holding off on this because the new reader is slower, pending
performance optimizations (which I'll do a little later on), however the
performance loss is of no practical consideration and only affects the
linker, which is still fast.

Therefore, it's better to get this old code out of the way to simplify
refactoring going forward.  In particular, I'm working on the diagnostic
system.

This is a little sad, in a way---this is some of my first Rust code that I'm
deleting.

DEV-10935
2022-04-11 16:11:49 -04:00
Mike Gerwitz f07c0e75be tamer: tameld (TameldError): Error sum type
This aggregates all non-panic errors that can occur during link time, making
`Box<dyn Error>` unnecessary.  I've been wanting to do this for a long time,
so it's nice seeing this come together.  This is a powerful tool, in that we
know, at compile time, all errors that can occur, and properly report on
them and compose them.  This method of error composition ensures that all
errors have a chance to be handled within their context, though it'll take
time to do so in a decent way.

This just maintains compatibility with the dynamic dispatch that was
previous occurring.  This work is being done to introduce the initial
diagnostic system, which was really difficult/confusing to do without proper
errors types at the top level, considering the toplevel is responsible for
triggering the diagnostic reporting.

The cycle error is in particular going to be interesting once the system is
in place, especially once it provides spans in the future, since it will
guide the user through the code to understand how the cycle formed.

More to come.

DEV-10935
2022-04-11 15:15:04 -04:00
Mike Gerwitz a1a4ad3e8e tamer: Introduce context into XirReader
tamec and tameld will now both introduce a `Context` to XIR, which will use
it to create spans.

Here's an example of an error, now that it's all working well together:

  $ target/release/tameld --emit xmle -o /dev/null path/to/package.xmlo
  error: invalid preproc:sym/@dim `9` at [/../path/to/package.xmlo offset 1175451-1175452]

A future task will make this human-readable by producing line and column
numbers, and perhaps even a snippet (if not now, then eventually).

It's exciting to see this coming together finally.

DEV-10934
2022-04-08 16:16:23 -04:00
Mike Gerwitz 2e3d94c3d6 tamer: obj::xmlo::reader: Simplify wip-xmlo-xir-reader flagging
This removes the flag from most of the code, which also resolves the
indentation.  Not only was it bothering me, but I don't want (a) every line
modified when the module body is hoisted and (b) `rustfmt` to reformat
everything when that happens.

This means that everything will be built, even though it's not used, when
the flag is off, but I see that as a good thing.

DEV-10863
2022-03-24 09:45:59 -04:00
Mike Gerwitz fbf786086a tamer: parse::Parser (lower_while_ok): New method
This introduces a WIP lowering operation, abstracting away quite a bit of
the manual wiring work, which is really important to providing an API that
provides the proper level of abstraction for actually understanding what the
system is doing.

This does not yet have tests associated with it---I had started, but it's a
lot of work and boilerplate for something that is going to
evolve.  Generally, I wouldn't use that as an excuse, but the robust type
definitions in play, combined with the tiny amount of actual logic, provide
a pretty high level of confidence.  It's very difficult to wire these types
together and produce something incorrect without doing something obviously
bad.

Similarly, I'm holding off on proper docs too, though I did write some
information here.

More to come, after I actually get to work on the XmloReader.

On a side note: I'm happy to have made progress on this, since this wiring
is something I've been dreading and wondering about since before the Parser
abstraction even existed.

Note also that this makes parser::feed_toks private again---I don't intend
to support push parsers yet, since they're only needed internally.  Maybe
for error recovery, but I'll wait to decide until it's actually needed.

DEV-10863
2022-03-23 14:31:16 -04:00
Mike Gerwitz b4a7591357 tamer: obj::xmlo::reader: Begin conversion to ParseState
This begins to transition XmloReader into a ParseState.  Unlike previous
changes where ParseStates were composed into a single ParseState, this is
instead a lowering operation that will take the output of one Parser and
provide it to another.

The mess in ld::poc (...which still needs to be refactored and removed)
shows the concept, which will be abstracted away.  This won't actually get
to the ASG in order to test that that this works with the
wip-xmlo-xir-reader flag on (development hasn't gotten that far yet), but
since it type-checks, it should conceptually work.

Wiring lowering operations together is something that I've been dreading for
months, but my approach of only abstracting after-the-fact has helped to
guide a sane approach for this.  For some definition of "sane".

It's also worth noting that AsgBuilder will too become a ParseState
implemented as another lowering operation, so:

  XIR -> XIRF -> XMLO -> ASG

These steps will all be streaming, with iteration happening only at the
topmost level.  For this reason, it's important that ASG not be responsible
for doing that pull, and further we should propagate Parsed::Incomplete
rather than filtering it out and looping an indeterminate number of times
outside of the toplevel.

One final note: the choice of 64 for the maximum depth is entirely
arbitrary and should be more than generous; it'll be finalized at some point
in the future once I actually evaluate what maximum depth is reasonable
based on how the system is used, with some added growing room.

DEV-10863
2022-03-22 14:06:52 -04:00
Mike Gerwitz 4c5b860195 tamer: Remove Ix generic from ASG
This is simply not worth it; the size is not going to be the bottleneck (at
least any time soon) and the generic not only pollutes all the things that
will use ASG in the near future, but is also incompatible with the SymbolId
default that is used everywhere; if we have to force it to 32 bits anyway,
then we may as well just default it right off the bat.

I thought that this seemed like a good idea at the time, and saving bits is
certainly tempting, but it was premature.
2022-01-14 10:21:49 -05:00
Mike Gerwitz 61f7a12975 tamer: xir::tree: Integrate AttrParserState into Stack
Note that AttrParse{r=>}State needs renaming, and Stack will get a better
name down the line too.  This commit message is accurate, but confusing.

This performs the long-awaited task of trying to observe, concretely, how to
combine two automata.  This has the effect of stitching together the state
machines, such that the union of the two is equivalent to the original
monolith.

The next step will be to abstract this away.

There are some important things to note here.  First, this introduces a new
"dead" state concept, where here a dead state is defined as an _accepting_
state that has no state transitions for the given input token.  This is more
strict than a dead state as defined in, for example, the Dragon Book, where
backtracking may occur.

The reason I chose for a Dead state to be accepting is simple: it represents
a lookahead situation.  It says, "I don't know what this token is, but I've
done my job, so it may be useful in a parent context".  The "I've done my
job" part is only applicable in an accepting state.

If the parser is _not_ in an accepting state, then an unknown token is
simply an error; we should _not_ try to backtrack or anything of the sort,
because we want only a single token of lookahead.

The reason this was done is because it's otherwise difficult to compose the
two parsers without requiring that AttrEnd exist in every XIR stream; this
has always been an awkward delimiter that was introduced to make the parser
LL(0), but I tried to compromise by saying that it was optional.  Of course,
I knew that decision caused awkward inconsistencies, I had just hoped that
those inconsistencies wouldn't manifest in practical issues.

Well, now it did, and the benefits of AttrEnd that we had in the previous
construction do not exist in this one.  Consequently, it makes more sense to
simply go from LL(0) to LL(1), which makes AttrEnd unnecessary, and a future
commit will remove it entirely.

All of this information will be documented, but I want to get further in
the implementation first to make sure I don't change course again and
therefore waste my time on docs.

DEV-11268
2021-12-16 09:44:02 -05:00
Mike Gerwitz 77c18d0615 tamer: xir: Remove Attr::Extensible
This removes XIRT support for attribute fragments.  The reason is that
because this is a write-only operation---fragments are used to concatenate
SymbolIds without reallocation, which can only happen if we are generating
XIR internally.

Given that this cannot happen during read, it was a mistake to complicate
the parsers.  But it makes sense why I did originally, given that the XIRT
parser was written for simplifying test cases.  But now that we want parsers
for real, and are writing production-quality parsers, this extra complexity
is very undesirable.

As a bonus, we also avoid any potential for heap allocations related to
attributes.  Granted, they didn't _really_ exist to begin with, but it was
part of XIRT, and was ugly.

DEV-11268
2021-12-06 14:26:58 -05:00
Mike Gerwitz f519dab2b6 tamer: xir::tree::attr::Attr::value_atom: Option<SymbolId>=>SymbolId
To maintain a proper abstraction, this cannot be the responsibility of the
caller; most callers should not know that fragments exist, letalone how to
handle them.
2021-11-16 12:41:03 -05:00
Mike Gerwitz 5233822322 tamer: xir: Remove Text enum
Like previous commits, this replaces the explicit escaping context with the
convention that all values retrieved from `xir` are unescaped on read and
escaped on write.

Comments are a notable TODO, since we must escape only `--`.

CData is also an issue.  I had _expected_ to use it as a means to avoid
unescaping fragments, but I had forgotten that quick_xml hard-codes escaping
on read, so that it can re-use BytesStart!  That is terribly unfortunate,
and may result in us having to re-implement our own read method in the
future to avoid this nonsense.  So I'm just leaving it as a TODO for now.

DEV-11081
2021-11-15 23:47:14 -05:00
Mike Gerwitz d710437ee4 tamer: xir::escape::CachingEscaper: New Escaper
As promised, this will cache previously seen escaped/unescaped values by
creating a two-way mapping between them.

DEV-11081
2021-11-15 16:44:24 -05:00
Mike Gerwitz 27ba03b59b tamer: xir::escape: Remove XirString in favor of Escaper
This rewrites a good portion of the previous commit.

Rather than explicitly storing whether a given string has been escaped, we
can instead assume that all SymbolIds leaving or entering XIR are unescaped,
because there is no reason for any other part of the system to deal with
such details of XML documents.

Given that, we need only unescape on read and escape on write.  This is
customary, so why didn't I do that to begin with?

The previous commit outlines the reason, mainly being an optimization for
the echo writer that is upcoming.  However, this solution will end up being
better---it's not implemented yet, but we can have a caching layer, such
that the Escaper records a mapping between escaped and unescaped SymbolIds
to avoid work the next time around.  If we share the Escaper between _all_
readers and the writer, the result is that

  1. Duplicate strings between source files and object files (many of which
     are read by both the linker and compiler) avoid re-unescaping; and
  2. Writers can use this cache to avoid re-escaping when we've already seen
     the escaped variant of the string during read.

The alternative would be a global cache, like the internment system, but I
did not find that to be appropriate here, since this is far less
fundamental and is much easier to compose.

DEV-11081
2021-11-12 14:03:23 -05:00
Mike Gerwitz b1c0783c75 tamer: xir::XirString: WIP implementation (likely going away)
I'm not fond of this implementation, which is why it's not fully
completed.  I wanted to commit this for future reference, and take the
opportunity to explain why I don't like it.

First: this task started as an idea to implement a third variant to
AttrValue and friends that indicates that a value is fixed, in the sense of
a fixed-point function: escaped or unescaped, its value is the same.  This
would allow us to skip wasteful escape/unescape operations.

In doing so, it became obvious that there's no need to leak this information
through the API, and indeed, no part of the system should care.  When we
read XML, it should be unescaped, and when we write, it should be
escaped.  The reason that this didn't quite happen to begin with was an
optimization: I'll be creating an echo writer in place of the current
filesystem-based copy in tamec shortly, and this would allow streaming XIR
directly from the reader to the writer without any unescaping or
re-escaping.

When we unescape, we know the value that it came from, so we could simply
store both symbols---they're 32-bit, so it results in a nicely compressed
64-bit value, so it's essentially cost-free, as long as we accept the
expense of internment.  This is `XirString`.  Then, when we want to escape
or unescape, we first check to see whether a symbol already exists and, if
so, use it.

While this works well for echoing streams, it won't work all that well in
practice: the unescaped SymbolId will be taken and the XirString discarded,
since nothing after XIR should be coupled with it.  Then, when we later
construct a XIR stream for writting, XirString will no longer be available
and our previously known escape is lost, so the writer will have to
re-escape.

Further, if we look at XirString's generic for the XirStringEscaper---it
uses phantom, which hints that maybe it's not in the best place.  Indeed,
I've already acknowledged that only a reader unescapes and only a writer
escapes, and that the rest of the system works with normal (unescaped)
values, so only readers and writers should be part of this process.  I also
already acknowledged that XirString would be lost and only the unescaped
SymbolId would be used.

So what's the point of XirString, then, if it won't be a useful optimization
beyond the temporary echo writer?

Instead, we can take the XirStringWriter and implement two caches on that:
mapping SymbolId from escaped->unescaped and vice-versa.  These can be
simple vectors, since SymbolId is a 32-bit value we will not have much
wasted space for symbols that never get read or written.  We could even
optimize for preinterned symbols using markers, though I'll probably not do
so, and I'll explain why later.

If we do _that_, we get even _better_ optimizations through caching that
_will_ apply in the general case (so, not just for echo), and we're able to
ditch XirString entirely and simply use a SymbolId.  This makes for a much
more friendly API that isn't leaking implementation details, though it
_does_ put an onus on the caller to pass the encoder to both the reader and
the writer, _if_ it wants to take advantage of a cache.  But that burden is
not significant (and is, again, optional if we don't want it).

So, that'll be the next step.
2021-11-10 12:22:10 -05:00
Mike Gerwitz 428d508be4 tamer: {ir::=>}{asg, xir}
See the previous commit.  There is no sense in some common "IR" namespace,
since those IRs should live close to whatever system whose data they
represent.

In the case of these, they are general IRs that can apply to many different
parts of the system.  If that proves to be a false statement, they'll be
moved.

DEV-10863
2021-11-04 16:13:27 -04:00
Mike Gerwitz cee6402f8b tamer: Move {ir::legacyir=>obj::xmlo::legacyir}
The IRs really ought to live where they are owned, especially given that
"IR" is so generic that it makes no sense for there to be a single location
for them; they're just data structures coupled with different phases of
compilation.

This will be renamed next commit; see that for details.

This also removes some documentation describing the lowering process,
because it's undergone a number of changes and needs to be accurately
re-summarized in another location.  That will come at a later time after the
work is further along so that I don't have to keep spending the time
rewriting it.

DEV-10863
2021-11-04 13:20:38 -04:00
Mike Gerwitz d045786cfb tamer: ir::xir::tree::Element::attrs: Wrap in Option
This allows AttrList not only to be lazily initialized (which is less of a
problem at the moment with Vec, but may become one in the future), but also
leaves a space open for attributes to be added _after_ having been
parsed.  It further leaves room to _take_ attributes from their `Element`.

This is important because the next commit will re-introduce the ability to
parse attributes independently, allowing us to put the parser in a state
where we can parse AttrList without an Element context.  To re-use that
parsing under an Element context, we can simply attach an AttrList after it
has been parsed.

Option adds no additional size cost to Vec, so we get this for free (except
for the tiny change that initializes the attribute list when we try to push
to it).

I also think this reads better ("attrs: None").  Though it makes the API
slightly more of a pain to work with.

DEV-10863
2021-10-29 16:34:05 -04:00
Mike Gerwitz 18ab032ba0 tamer: Begin XIR-based xmlo reader impl
There isn't a whole lot here, but there is additional work needed in various
places to support upcoming changes and so I want to get this commited to
ease the cognitive burden of what I have thusfar.  And to stop stashing.  We
have a feature flag for a reason.

DEV-10863
2021-10-28 21:21:30 -04:00
Mike Gerwitz 581b9d4e65 tamer: Use `..` for tuple unimportant variant matches
Tbh, I was unaware that this was supported by tuple variants until reading
over the Rustc source code for something.  (Which I had previously read, but
I must have missed it.)

This is more proper, in the sense that in a lot of cases we not only care
about how many values a tuple has, but if we explicitly match on them using
`_`, then any time we modify the number of values, it would _break_ any code
doing so.  Using this method, we improve maintainability by not causing
breakages under those circumstances.

But, consequently, it's important that we use this only when we _really_
don't care and don't want to be notified by the compiler.

I did not use `..` as a prefix, even where supported, because the intent is
to append additional information to tuples.  Consequently, I also used `..`
in places where no additional fields currently exist, since they may in the
future (e.g. introducing `Span` for `IdentObject`).
2021-10-15 12:28:59 -04:00
Mike Gerwitz 739cf7e6eb tamer: ir::asg::object::IdentObject: Define methods from IdentObjectData
In particular, `name` needn't return an `Option`.  `fragment` also returns a
copy, since it's just a `SymbolId`.  (It really ought to be a newtype rather
than an alias, but we'll worry about that some other time.)

These changes allow us to remove some runtime panics.

DEV-10859
2021-10-14 14:38:02 -04:00
Mike Gerwitz f055cb77c2 tamer: ld::xmle: Narrow Sections types
This moves the logic that sorts identifiers into sections into Sections
itself, and introduces XmleSections to allow for mocking for testing.

This then allows us to narrow the types significantly, eliminating some
runtime checks.  The types can be narrowed further, but I'll be limiting the
work I'll be doing now; this'll be inevitably addressed as we use the ASG
for the compiler.

This also handles moving Sections tests, which was a TODO from the previous
commit.

DEV-10859
2021-10-14 12:40:13 -04:00
Mike Gerwitz ea11cf1416 tamer: ld::xmle::lower: Extract sectioning into Sections
This is the appropriate place to be, now that we've begun narrowing the
types.  We'll be able to do so further; this is just the first step.

This does not yet move the tests, but the code is still tested because it's
tightly coupled with `sort`.  Those will move in the next commit(s).

DEV-10859
2021-10-12 12:15:11 -04:00
Mike Gerwitz 08d92ca663 tamer: ld::xmle::sections: Remove generic object type
xmle sections will only ever contain an object of one type, so there is no
use in making this generic.

I think the original plan was to have this represent, generically, sections
of some object file (like ELF), but doing so would require a significant
redesign anyway, so it makes no sense.  This is easier to reason about.

DEV-10859
2021-10-12 10:35:14 -04:00
Mike Gerwitz df328da71f tamer: ir::asg::SortableAsg: Move into ld::xmle::lower
This has always been a lowering operation, but it was not phrased in terms
of it, which made the process a bit more confusing to understand.

The implementation hasn't changed, but this is an incremental refactoring
and so exposes BaseAsg and its `graph` field temporarily.

DEV-10859
2021-10-12 09:49:33 -04:00