Commit Graph

850 Commits (dd6a6dd196a394e6c3ff919e3e9c9514a5ad2613)

Author SHA1 Message Date
Mike Gerwitz 4cb478a42d tamer: parser::ParseState::delegate_lookahead: New concept
This introduces a new method similar to the previous `delegate`, but with
another closure that allows for handling lookahead tokens from the child
parser.

Admittedly, this isn't exactly what I was going for---a list of arguments
isn't exactly self-documenting, especially with the brevity when the
arguments line up---but this was easy to do and so I'll run with this for
now.

This also modified `delegate` to accept a context, even though it wasn't
necessary, both for consistency with its lookup counterpart and for brevity
with the `into` argument (allowing, in our case, to just pass the name of
the variant, rather than a closure).

I'm not going to handle the actual starting and accepting state stitching
abstraction for now; I'd like to observe future boilerplate more before I
consider the best way to handle it, though I do have some ideas.

DEV-10863
2022-03-29 14:46:43 -04:00
Mike Gerwitz 2a3d5be159 tamer: parse::ParseState::delegate: Initial state stitching concept
This is the delegation portion of what I've come to call "state
stitching"---wiring together two state machines that recognize the same
input tokens.

This handles the delegation of tokens once the parser has been entered, but
does not yet handle the actual stitching part of it: wiring the start and
accepting states of the child parser to the parent.

This is indirectly tested by the XmloReader, but it will receive its own
tests once I further finalize this concept.  I'm playing around with some
ideas.  With that said, a quick visual inspection together with the
guarantees provided by the type system should convince any familiar reader
of its correctness.

DEV-10863
2022-03-29 14:12:26 -04:00
Mike Gerwitz df05a71508 tamer: obj::xmlo::reader: Emphasize generic SymtableState stitching for Object
This simply makes the block more generic to emphasize how it can be
abstracted away.

DEV-10863
2022-03-29 11:25:05 -04:00
Mike Gerwitz f42288f3a2 tamer: obj::xmlo::reader: Begin symbol table parsing
This wasn't the simplest thing to start with, but I wanted to explore
something with a higher level of complexity.  There is some boilerplate to
observe here, including:

  1. The state stitching (as I guess I'm calling it now) of SymtableState
     with XmloReaderState is all boilerplate and requires no lookahead,
     presenting an abstraction opportunity that I was holding off on
     previously (attr parsing for XIRF requires lookahead).
  2. This is simply collecting attributes into a struct.  This can be
     abstracted away in the future.
  3. Creating stub parsers to verify that generics are stitched rather than
     being tightly coupled with another state is boilerplate that maybe can
     be abstracted away after a pattern is observed in future tests.

DEV-10863
2022-03-29 11:14:47 -04:00
Mike Gerwitz f402e51d04 tamer: parse: More flexible Transition API
This does some cleanup and adds `parse::Object` for use in disambiguating
`From` for `ParseStatus`, allowing the `Transition` API to be much more
flexible in the data it accepts and automatically converts.  This allows us
to concisely provide raw output data to be wrapped, or provide `ParseStatus`
directly when more convenient.

There aren't yet examples in the docs; I'll do so once I make sure this API
is actually utilized as intended.

DEV-10863
2022-03-25 16:45:32 -04:00
Mike Gerwitz c0fa89222e tamer: obj::xmlo::ir::Dim: New enum
This replaces u8 and will be used for the new XmloReader.

Previously I wasn't sure what direction TAMER was going to go in with
regards to dimensionality, but I do not expect that higher dimensions will
be supported, and if they are, they'd very likely compile down to lower ones
and create an illusion of higher-dimensionality.

Whatever the future holds, it's not used today, and I'd rather these types
be correct.

ASG needs changing too, but one step at a time.

DEV-10863
2022-03-25 14:28:18 -04:00
Mike Gerwitz 279ddc79d7 tamer: parse::TransitionResult: Alias=>newtype
This converts the tuple type alias into a newtype, so that we may provide
our own implementations.

This differs from a previous approach that I took, which involved making
this type `Result<(S, T), (S, E)>` so that the return values composed well
with other functions.  But the reality is that this is used only by other
`ParseState`s and `Parser`, so it's unnecessary.

However, this is also an attempt to utilize the new Try and FromResidual
traits; note how the Try associated types match precisely what I was trying
to do before, though they're used as intermediate types.  I'll see how this
evolves.

DEV-10863
2022-03-25 12:28:50 -04:00
Mike Gerwitz 2e98a69d15 Revert "tamer: parse::TransitionResult: Move common Transition into Result"
This reverts commit bf5da75096.
2022-03-25 09:17:25 -04:00
Mike Gerwitz bf5da75096 tamer: parse::TransitionResult: Move common Transition into Result
This allows the Results to compose and, importantly, is compatible with
`?` without having to put in any extra effort.

This makes puts the caller in an awkward spot, so I introduced a utility
function `result_tup0_invert` for now; we'll see if that stays or evolves
differently.

DEV-10863
2022-03-24 23:48:30 -04:00
Mike Gerwitz 9d9b1f30a8 tamer: obj::xmlo::reader: Move XmloEvent to top of module
Since this is the object produced by this parser, this is likely the most
useful first thing to present as a summary of what `XmloReader` actually
does.

DEV-10863
2022-03-24 10:14:40 -04:00
Mike Gerwitz 2e3d94c3d6 tamer: obj::xmlo::reader: Simplify wip-xmlo-xir-reader flagging
This removes the flag from most of the code, which also resolves the
indentation.  Not only was it bothering me, but I don't want (a) every line
modified when the module body is hoisted and (b) `rustfmt` to reformat
everything when that happens.

This means that everything will be built, even though it's not used, when
the flag is off, but I see that as a good thing.

DEV-10863
2022-03-24 09:45:59 -04:00
Mike Gerwitz fab7b16ea0 tamer: obj::xmlo::reader: Parse package attributes
Finally we get to do some actual parsing with all of the preparatory work!

This means that we're finally ready to fully replace the old XmloReader,
provided that I'm okay with some boilerplate / lack of abstractions for
now (and I am, because all I've been doing is working on abstractions to
prepare lowering operations).

DEV-10863
2022-03-23 16:48:51 -04:00
Mike Gerwitz ad8616aaa1 tamer: xir::attr::Attr: Convert to tuple struct with public fields
This makes more sense for pattern matching.  Encapsulation of these fields
is not necessary, given that it's passed around as an owned value and its
`new` method constructs it verbatim; the individual fields are
self-validating.

DEV-10863
2022-03-23 16:41:28 -04:00
Mike Gerwitz fbf786086a tamer: parse::Parser (lower_while_ok): New method
This introduces a WIP lowering operation, abstracting away quite a bit of
the manual wiring work, which is really important to providing an API that
provides the proper level of abstraction for actually understanding what the
system is doing.

This does not yet have tests associated with it---I had started, but it's a
lot of work and boilerplate for something that is going to
evolve.  Generally, I wouldn't use that as an excuse, but the robust type
definitions in play, combined with the tiny amount of actual logic, provide
a pretty high level of confidence.  It's very difficult to wire these types
together and produce something incorrect without doing something obviously
bad.

Similarly, I'm holding off on proper docs too, though I did write some
information here.

More to come, after I actually get to work on the XmloReader.

On a side note: I'm happy to have made progress on this, since this wiring
is something I've been dreading and wondering about since before the Parser
abstraction even existed.

Note also that this makes parser::feed_toks private again---I don't intend
to support push parsers yet, since they're only needed internally.  Maybe
for error recovery, but I'll wait to decide until it's actually needed.

DEV-10863
2022-03-23 14:31:16 -04:00
Mike Gerwitz b4a7591357 tamer: obj::xmlo::reader: Begin conversion to ParseState
This begins to transition XmloReader into a ParseState.  Unlike previous
changes where ParseStates were composed into a single ParseState, this is
instead a lowering operation that will take the output of one Parser and
provide it to another.

The mess in ld::poc (...which still needs to be refactored and removed)
shows the concept, which will be abstracted away.  This won't actually get
to the ASG in order to test that that this works with the
wip-xmlo-xir-reader flag on (development hasn't gotten that far yet), but
since it type-checks, it should conceptually work.

Wiring lowering operations together is something that I've been dreading for
months, but my approach of only abstracting after-the-fact has helped to
guide a sane approach for this.  For some definition of "sane".

It's also worth noting that AsgBuilder will too become a ParseState
implemented as another lowering operation, so:

  XIR -> XIRF -> XMLO -> ASG

These steps will all be streaming, with iteration happening only at the
topmost level.  For this reason, it's important that ASG not be responsible
for doing that pull, and further we should propagate Parsed::Incomplete
rather than filtering it out and looping an indeterminate number of times
outside of the toplevel.

One final note: the choice of 64 for the maximum depth is entirely
arbitrary and should be more than generous; it'll be finalized at some point
in the future once I actually evaluate what maximum depth is reasonable
based on how the system is used, with some added growing room.

DEV-10863
2022-03-22 14:06:52 -04:00
Mike Gerwitz f6957ff028 tamer: parse::Parser: Extract logic from Iterator impl
This introduces a (still-private) way to _push_ tokens into the parser,
rather than relying purely on a pull-based interface.  Not only does this
simplify the iterator, but this is also preparing to make the new `feed_tok`
public so that parsers can be composed in more contexts.  I suspect that
this method may also be useful for error recovery, since it can be used to
inject tokens into arbitrary points of a token stream.

I kept the new method private for now so that I can introduce the new API
and docs separate from this refactoring.

DEV-10863
2022-03-22 10:10:59 -04:00
Mike Gerwitz ceb00c4df5 tamer: xir: Complete parse type migration
A previous commit moved the parser.  This updates the types so that they can
actually be utilized in that context.

DEV-10863
2022-03-21 15:50:43 -04:00
Mike Gerwitz 14638a612f tamer: {xir::=>}parse: Move parser out of XIR
The parsing framework originally created for XIR is now more general and
useful to other things.  We'll see how this evolves.

This needs additional documentation, but I'd like to see how it changes as
I implement XmloReader and then some of the source readers first.

DEV-10863
2022-03-18 16:24:53 -04:00
Mike Gerwitz 0360226caa tamer: xir::parse: Generalize input token type
This adds a `Token` type to `ParseState`.  Everything uses `xir::Token`
currently, but `XmloReader` will use `xir::flat::Object`.

Now that this has been generalized beyond XIR, the parser ought to be
hoisted up a level.

DEV-10863
2022-03-18 15:26:05 -04:00
Mike Gerwitz 150b3b9aa4 tamer: xir::flat: Improve parser validation
This does a couple of things: it ensures that documents one and only one
root note, and it properly handles dead transitions once parsing is
complete (allowing it to be composed).

This should make XIRF feature-complete for the time being.  It does rely on
the assumption that the reader is stripping out any trailing whitespace, so
I guess we'll see if that's true as we proceed.

DEV-10863
2022-03-17 23:22:38 -04:00
Mike Gerwitz f04d845452 tamer: xir::flat::parse_token: Remove now-unapplicable comment
Forgot to delete this in a previous commit.

DEV-10863
2022-03-17 21:37:05 -04:00
Mike Gerwitz aba89f809d tamer: xir::parse: UnexpectedEof Span at final offset
I'm not rendering errors yet in practice, so this wouldn't have been
noticed, but we want error messages to reference the final byte in a file on
EOF, not the offset of the last-encountered token, which would be confusing.

This doesn't _directly_ pertain to what I'm working on; I just happened to
notice it.

DEV-10863
2022-03-17 21:33:05 -04:00
Mike Gerwitz e18eb2a4ac tamer: xir::flat::State::parse_node: Use TransitionResult
This was simply missed in a previous commit.

DEV-10863
2022-03-17 16:30:35 -04:00
Mike Gerwitz 6b8f0663ea tamer: xir::{tree::=>}attr: Move
With the introduction of XIRF, attribute parsing is no longer a XIRT thing.

DEV-10863
2022-03-17 16:10:56 -04:00
Mike Gerwitz 7b6d68af85 tamer: xir::parse::Transition: Generalize flat::Transition
XIRF introduced the concept of `Transition` to help document code and
provide mental synchronization points that make it easier to reason about
the system.  I decided to hoist this into XIR's parser itself, and have
`parse_token` accept an owned state and require a new state to be returned,
utilizing `Transition`.

Together with the convenience methods introduced on `Transition` itself,
this produces much clearer code, as is evidenced by tree::Stack (XIRT's
parser).  Passing an owned state is something that I had wanted to do
originally, but I thought it'd lead to more concise code to use a mutable
reference.  Unfortunately, that concision lead to code that was much more
difficult than necessary to understand, and ended up having a net negative
benefit by leading to some more boilerplate for the nested types (granted,
that could have been alleviated in other ways).

This also opens up the possibility to do something that I wasn't able to
before, which was continue to abstract away parser composition by stitching
their state machines together.  I don't know if this'll be done immediately,
but because the actual parsing operations are now able to compose
functionally without mutability getting the way, the previous state coupling
issues with the parent parser go away.

DEV-10863
2022-03-17 16:02:05 -04:00
Mike Gerwitz 899fa79e59 tamer: xir::flat: Initial XIRF implementation
This introduces XIR Flat (XIRF), which is conceptually between XIR and
XIRT.  This provides a more appropriate level of abstraction for further
lowering operations to parse against, and removes the need for other parsers
to perform their own validations (inappropriately) to ensure well-formed
XML.

There is still some cleanup worth doing, including moving some of the
parsing responsibility up a level back into the XIR parser.

DEV-10863
2022-03-17 13:08:16 -04:00
Mike Gerwitz ce48a654b1 tamer: span::Span::offset_add: Make const
This behavior is unchanged, but it allows us to create more constant spans
for testing.  For example:

  const S = DUMMY_SPAN.offset_add(1).unwrap();

This, in turn, will allow for removing lazy_static! for tests that use it
for span generation.

DEV-10863
2022-03-16 14:16:28 -04:00
Mike Gerwitz 18cb5e7b39 tamer: Update dependencies
Petgraph was previously held back due to petgraph-graphml.  I'd like to
transition away from that at some point, given that it's tied to petgraph
and also pulls in xmlns, on top of quick-xml and our XIR, but that can come
down the line.
2022-03-11 10:51:51 -05:00
Mike Gerwitz 2f703ab2df tamer: obj::xmlo: Remove PackageAttrs in favor of token stream
The Options here are awkward and will be able to go away in the new reader
and in AsgBuilder once it has a proper state machine.

This gets rid of some of the initial migratory work for the new reader,
because PackageAttrs is gone.  I'm going to wait to update this to the new
way until I get further into this.

DEV-11449
2022-03-10 15:44:54 -05:00
Mike Gerwitz d428755a2e tamer: obj::xmlo::XmloEvent::SymDeps: Remove
This is not longer needed after the previous commit.
2022-03-10 13:43:07 -05:00
Mike Gerwitz dcfae8a624 tamer: obj::xmlo: Begin transition to streaming quick-xml reader
I'm finally back to TAMER development.

The original plan, some time ago, was to gate an entirely new XmloReader
behind a feature flag (wip-xmlo-xir-reader), and go from there, leaving the
existing implementation untouched.  Unfortunately, it became too difficult
and confusing to marry the old aggregate API with the new streaming one.

AsgBuilder is the only system interacting with XmloReader, so I decided (see
previous commits) to just go the route of refactoring the existing
one.  I'm not yet sure if I'll continue to progressively refactor this one
and eliminate the two separate implementations behind the flag, or if I'll
get this API similar and then keep the flag and reimplement it.  But I'll
know soon.

DEV-11449
2022-03-10 13:31:24 -05:00
Mike Gerwitz 74ddc77adb tamer: xir::escape::CachingEscaper: allow(dead_code) for feature-flagged code
For now, until this feature flag is removed, so that we do not see warnings
when the flag is off.
2022-03-10 10:03:07 -05:00
Mike Gerwitz 76b16fed09 tamer: iter::collect::TryCollect::try_collect_ok: Disambiguate try_collect
The Rust team has begun to introduce try_collect.  I will keep an eye on
this implementation and revisit this, but for the time being, I'm going to
disambiguate this so that I can move on without worrying about a future
breakage.

  - https://github.com/rust-lang/rust/issues/94047
  - https://doc.rust-lang.org/nightly/std/iter/trait.Iterator.html#method.try_collect
2022-03-08 12:55:54 -05:00
Mike Gerwitz 4c5b860195 tamer: Remove Ix generic from ASG
This is simply not worth it; the size is not going to be the bottleneck (at
least any time soon) and the generic not only pollutes all the things that
will use ASG in the near future, but is also incompatible with the SymbolId
default that is used everywhere; if we have to force it to 32 bits anyway,
then we may as well just default it right off the bat.

I thought that this seemed like a good idea at the time, and saving bits is
certainly tempting, but it was premature.
2022-01-14 10:21:49 -05:00
Mike Gerwitz 5af698d15c tamer: xir::{tree::=>}parse: Move module
It's a bit odd that I've done next to nothing with TAMER for the past week
or so, and decided to do this one small thing before I go on break for the
holidays, but I felt compelled to do _something_.  Besides, this gets me in
a better spot for the inevitable mental planning and writing I'll be doing
over the holidays.

This move was natural, given what this has evolved into---it has nothing to
do with the concept of a "tree", and the modules imports emphasized that
fact given the level of inappropriate nesting.
2021-12-23 13:17:18 -05:00
Mike Gerwitz 8221e3a011 tamer: xir::tree::Stack: Refactor transitions
Now that the parser has been simplified by removing attributes, we can
further simplify the state transitions to make it more clear what further
refactoring can be done.

DEV-11339
2021-12-17 11:40:30 -05:00
Mike Gerwitz d5a2d43526 tamer: xir::tree::attr::parse::AttrParse{r=>}State
Simply correcting a naming inconsistency between the trait and the concrete
type.

DEV-11339 / DEV-11268
2021-12-17 10:22:29 -05:00
Mike Gerwitz 0cc0bc9d5a tamer: xir::Token::AttrEnd: Remove
More information can be found in the prior commit message, but I'll
summarize here.

This token was introduced to create a LL(0) parser---no tokens of
lookahead.  This allowed the underlying TokenStream to be freely passed to
the next system that needed it.

Since then, Parser and ParseState were introduced, along with
ParseStatus::Dead, which introduces the concept of lookahead for a single
token---an LL(1) grammar.

I had always suspected that this would happen, given the awkwardness of
AttrEnd; it was just a matter of time before the right abstraction
manifested itself to handle lookahead.

DEV-11339
2021-12-17 10:14:31 -05:00
Mike Gerwitz 61f7a12975 tamer: xir::tree: Integrate AttrParserState into Stack
Note that AttrParse{r=>}State needs renaming, and Stack will get a better
name down the line too.  This commit message is accurate, but confusing.

This performs the long-awaited task of trying to observe, concretely, how to
combine two automata.  This has the effect of stitching together the state
machines, such that the union of the two is equivalent to the original
monolith.

The next step will be to abstract this away.

There are some important things to note here.  First, this introduces a new
"dead" state concept, where here a dead state is defined as an _accepting_
state that has no state transitions for the given input token.  This is more
strict than a dead state as defined in, for example, the Dragon Book, where
backtracking may occur.

The reason I chose for a Dead state to be accepting is simple: it represents
a lookahead situation.  It says, "I don't know what this token is, but I've
done my job, so it may be useful in a parent context".  The "I've done my
job" part is only applicable in an accepting state.

If the parser is _not_ in an accepting state, then an unknown token is
simply an error; we should _not_ try to backtrack or anything of the sort,
because we want only a single token of lookahead.

The reason this was done is because it's otherwise difficult to compose the
two parsers without requiring that AttrEnd exist in every XIR stream; this
has always been an awkward delimiter that was introduced to make the parser
LL(0), but I tried to compromise by saying that it was optional.  Of course,
I knew that decision caused awkward inconsistencies, I had just hoped that
those inconsistencies wouldn't manifest in practical issues.

Well, now it did, and the benefits of AttrEnd that we had in the previous
construction do not exist in this one.  Consequently, it makes more sense to
simply go from LL(0) to LL(1), which makes AttrEnd unnecessary, and a future
commit will remove it entirely.

All of this information will be documented, but I want to get further in
the implementation first to make sure I don't change course again and
therefore waste my time on docs.

DEV-11268
2021-12-16 09:44:02 -05:00
Mike Gerwitz 0c7f04e092 tamer: xir::tree: Simplify Stack and remove isolated attr remnants
These were missed from a couple of commits ago, after I recalled that I
could now simplify the Stack variants; they were made more complicated due
to isolated attribute parsing.

These progressive refactorings do a good job illustrating why composing
parsers is better than a monolith---the complexity of the parsers is
significantly reduced, and the number of combinations of states are also
greatly reduced, which allows us to reason about them in isolation.

DEV-11268
2021-12-14 12:49:06 -05:00
Mike Gerwitz 0061a13d63 tree: xir::tree::Object: Remove now-unneeded enum
This was added only for isolated attribute parsing.  Of course, this does
mean that a new union type will be needed when combining the two parsers,
depending on the desired resolution, but that'll come at a later time and
possibly in a more general way.

DEV-11268
2021-12-14 12:44:32 -05:00
Mike Gerwitz c7f846752d tamer: xir::tree: Remove now-unused isolated attribute parsing
This is handled by the new AttrState, so this is largely just removing
now-duplicate code.

DEV-11268
2021-12-14 12:42:02 -05:00
Mike Gerwitz 69acba3ec0 tamer: xir::tree: Use parse::Parser for parse
All tree module parsing functions now make use of parse::Parser.

This module will eventually be hoisted from tree.

DEV-11268
2021-12-14 12:36:35 -05:00
Mike Gerwitz b30d7dc84e tamer: xir::tree::parser_from: Use parse::Parser
This nearly completely integrates the new Parser with xir::tree, but does
not yet compose AttrParseState.  I also need to determine what to do with
`parse()` and, further, make `parser_from` generic as part of mod parse.

If we take a moment to reflect on all of the changes, this struggle has been
a roundabout way of converting tree's parser into parse::Parser; providing
a trait for Stack (as ParseState); beginning parser decomposition; and
moving some common logic into Parser.  The composition of parsers is the
final piece to be realized.

This could have been a lot less work if I really understood exactly what I
wanted to do up front, but as was mentioned in previous commits, I was
really confusing myself trying to maintain API BC in ways that I should not
have for XmloReader.  More on that will be coming soon as well.

DEV-11268
2021-12-13 16:57:04 -05:00
Mike Gerwitz 6e9d139373 tamer: xir::tree::parse::Parser: Remove lifetime
This will allow Parser to operate on both owned and &mut values, and is the
same approach that Rust's built-in iterators take.

This is at first quite surprising, and I often forget that this is a
feature, and, as a bonus, an attractive way to avoid lifetimes in struct
definitions when generics are used for the type that may become a
reference.

DEV-11268
2021-12-13 16:51:15 -05:00
Mike Gerwitz f09900b80c tamer: xir::tree: Remove isolated AttrList parsing
This isn't currently used by anything, and this is collecting, which does
not fit well with the streaming model.  AttrList was originally written for
Element parsing, and the isolated attr parser was written for test cases,
before it was fully decided how this system ought to work.

Instead, if AttrList is in fact needed, we can either collect (ideally not)
or implement Extend for AttrList.  (Or create TryExtend.)

DEV-11268
2021-12-13 16:20:50 -05:00
Mike Gerwitz 29fdf5428c tamer: xir::tree: {Parse=>Stack}Error
Prepare to adopt parse::ParseError, which will contain StackError.

DEV-11268
2021-12-13 15:27:20 -05:00
Mike Gerwitz faed32af7e tamer: xir::tree::ParserState: Remove and expose Stack directly
This removes the layer of encapsulation that was hiding Stack, which is the
actual parser.  The new layer of encapsulation is parse::Parser, which will
be introduced here soon.  Baby steps, so it's clear how this evolves.

DEV-11268
2021-12-13 15:02:08 -05:00
Mike Gerwitz 24e9b94b37 tamer: xir::tree::Parsed: Remove in favor of xir::tree::parse::Parsed
These were the same thing after the previous commit.  This moves toward
tree::Stack becoming a ParseState.

DEV-11268
2021-12-13 14:29:16 -05:00
Mike Gerwitz 48517502d9 tamer: xir::tree::Parsed: Mirror xir::tree::parse::Parsed
I think it's obvious where the next commit is going---replace
xir::tree::Parsed.

DEV-11268
2021-12-13 14:19:12 -05:00
Mike Gerwitz c6d6f44bcb tamer: xir::tree::parse: ParseStatus and Parsed
The old Parsed was renamed to ParseStatus to be used by Parser, and Parser
converts it into Parsed, which has the same variants as it did before and
has all but the Done variant, since it's not possible for Parser to yield
it.

DEV-11268
2021-12-10 16:51:53 -05:00
Mike Gerwitz 9facc26b4f tamer: xir::tree::parse: Use new Parsed::Done variant over None
This removes Option from ParseState, as mentioned in previous commits.

This is ideal because it not only removes a layer of abstraction, but also
makes the intent very clear; the use of None was too tied to the concept of
an Iterator, which is the concern of Parser, _not_ ParseState.

This is now similar to tree::Parsed, which will help with that refactoring
shortly.

The Done variant is not accessible outside of Parser, since it always
coverts it to None (to halt iteration); given that, we should have another
public-facing type, as was also mentioned in a previous commit.

DEV-11268
2021-12-10 16:22:02 -05:00
Mike Gerwitz 38363da9ff tamer: xir::tree: {TokenStream=>ParseState}
This also renames related types.

See previous commits for more in formation.  In essence, this trait
represents the reification of all parser state.  The omission of "r" in the
name ParseState is intentional, since it indicates the state of a current
parse.  We'll see whether that naming ends up being too confusing; it's easy
enough to change.

DEV-11268
2021-12-10 15:42:01 -05:00
Mike Gerwitz 8eddf2f5ef tamer: xir::tree::parse: Remove TokenStreamParser trait
This just leaves Parser, which is what I started with, but I wasn't sure how
far I was going to take this.  I went against my usual judgment in creating
a trait that I may not need, in an attempt to try to reason about the API
that I wanted, because it wasn't yet clear at the time whether the Parser
ought to be generic.

Since then (as detailed in the last commit), this has become more of a
coordinator/mediator, and the real parser is actually TokenStreamState,
which will be renamed shortly.

DEV-11268
2021-12-10 14:58:44 -05:00
Mike Gerwitz bfe46be5bb tamer: xir::tree::attr_parser_from: Integrate AttrParser
This begins to integrate the isolated AttrParser.  The next step will be
integrating it into the larger XIRT parser.

There's been considerable delay in getting this committed, because I went
through quite the struggle with myself trying to determine what balance I
want to strike between Rust's type system; convenience with parser
combinators; iterators; and various other abstractions.  I ended up being
confounded by trying to maintain the current XmloReader abstraction, which
is fundamentally incompatible with the way the new parsing system
works (streaming iterators that do not collect or perform heap
allocations).

There'll be more information on this to come, but there are certain things
that will be changing.

There are a couple problems highlighted by this commit (not in code, but
conceptually):

  1. Introducing Option here for the TokenParserState doesn't feel right, in
     the sense that the abstraction is inappropriate.  We should perhaps
     introduce a new variant Parsed::Done or something to indicate intent,
     rather than leaving the reader to have to read about what None actually
     means.
  2. This turns Parsed into more of a statement influencing control
     flow/logic, and so should be encapsulated, with an external equivalent
     of Parsed that omits variants that ought to remain encapsulated.
  3. TokenStreamState is true, but these really are the actual parsers;
     TokenStreamParser is more of a coordinator, and helps to abstract away
     some of the common logic so lower-level parsers do not have to worry
     about it.  But calling it TokenStreamState is both a bit
     confusing and is an understatement---it _does_ hold the state, but it
     also holds the current parsing stack in its variants.

Another thing that is not yet entirely clear is whether this AttrParser
ought to care about detection of duplicate attributes, or if that should be
done in a separate parser, perhaps even at the XIR level.  The same can be
said for checking for balanced tags.  By pushing it to TokenStream in XIR,
we would get a guaranteed check regardless of what parsers are used, which
is attractive because it reduces the (almost certain-to-otherwise-occur)
risk that individual parsers will not sufficiently check for semantically
valid XML.  But it does _potentially_ match error recovery more
complicated.  But at the same time, perhaps more specific parsers ought not
care about recovery at that level.

Anyway, point being, more to come, but I am disappointed how much time I'm
spending considering parsing, given that there are so many things I need to
move onto.  I just want this done right and in a way that feels like it's
working well with Rust while it's all in working memory, otherwise it's
going to be a significant effort to get back into.

DEV-11268
2021-12-10 14:25:08 -05:00
Mike Gerwitz 0e08cf3efe tamer: xir::tree::parse: EOF span
This stores the last seen Span and uses that when reporting EOF, so that the
user will be able to be notified of where exactly the problem occurred.

When I get into creating combinators, it'll be the responsibility of those
combinators to ensure that any None return value will be supplemented by its
own last span.

DEV-11268
2021-12-06 15:34:29 -05:00
Mike Gerwitz 325c3167ee tamer: xir::Token::span: New method
This permits retrieving a Span from any Token variant.  To support this,
rather than having this return an Option, Token::AttrEnd was augmented with
a Span; this results in a much simpler and friendlier API.

DEV-11268
2021-12-06 14:48:55 -05:00
Mike Gerwitz 77c18d0615 tamer: xir: Remove Attr::Extensible
This removes XIRT support for attribute fragments.  The reason is that
because this is a write-only operation---fragments are used to concatenate
SymbolIds without reallocation, which can only happen if we are generating
XIR internally.

Given that this cannot happen during read, it was a mistake to complicate
the parsers.  But it makes sense why I did originally, given that the XIRT
parser was written for simplifying test cases.  But now that we want parsers
for real, and are writing production-quality parsers, this extra complexity
is very undesirable.

As a bonus, we also avoid any potential for heap allocations related to
attributes.  Granted, they didn't _really_ exist to begin with, but it was
part of XIRT, and was ugly.

DEV-11268
2021-12-06 14:26:58 -05:00
Mike Gerwitz 42b5007402 tamer: xir:tree: Begin work on composable XIRT parser
The XIRT parser was initially written for test cases, so that unit tests
should assert more easily on generated token streams (XIR).  While it was
planned, it wasn't clear what the eventual needs would be, which were
expected to differ.  Indeed, loading everything into a generic tree
representation in memory is not appropriate---we should prefer streaming and
avoiding heap allocations when they’re not necessary, and we should parse
into an IR rather than a generic format, which ensures that the data follow
a proper grammar and are semantically valid.

When parsing attributes in an isolated context became necessary for the
aforementioned task, the state machine of the XIRT parser was modified to
accommodate.  The opposite approach should have been taken---instead of
adding complexity and special cases to the parser, and from a complex parser
extracting a simple one (an attribute parser), we should be composing the
larger (full XIRT) parser from smaller ones (e.g. attribute, child
elements).

A combinator, when used in a functional sense, refers not to combinatory
logic but to the composition of more complex systems from smaller ones.  The
changes made as part of this commit begin to work toward combinators, though
it's not necessarily evident yet (to you, the reader) how that'll work,
since the code for it hasn't yet been written; this is commit is simply
getting my work thusfar introduced so I can do some light refactoring before
continuing on it.

TAMER does not aim to introduce a parser combinator framework in its usual
sense---it favors, instead, striking a proper balance with Rust’s type
system that permits the convenience of combinators only in situations where
they are needed, to avoid having to write new parser
boilerplate.  Specifically:

  1. Rust’s type system should be used as combinators, so that parsers are
  automatically constructed from the type definition.

  2. Primitive parsers are written as explicit automata, not as primitive
     combinators.

  3. Parsing should directly produce IRs as a lowering operation below XIRT,
     rather than producing XIRT itself.  That is, target IRs should consume
     XIRT and produce parse themselves immediately, during streaming.

In the future, if more combinators are needed, they will be added; maybe
this will eventually evolve into a more generic parser combinator framework
for TAME, but that is certainly a waste of time right now.  And, to be
honest, I’m hoping that won’t be necessary.
2021-12-06 11:27:39 -05:00
Mike Gerwitz fd1b1527d6 tamer: Remove tests invoking cargo and associated libs
There are a number of reasons for this, where the benefits do not make up
for the losses.

First: this is actually invoking cargo.  Not only is this not necessary, but
it's not desirable: cargo by default hits the network and does all sorts of
other stuff, when all we want to do is invoke the executable.  So the tests
aren't really testing the right thing in that sense.  See the previous
commit for more information.

The way it invokes cargo is different than the way the Makefile invokes
cargo, so on my system, it's actually invoking a _different cargo_!  This is
causing problems, in particular with lock files, which causes my tests to
fail.

Importantly, this also removes a _lot_ of dependencies, which removes a lot
of supplier chain risk and a lot of code to audit.  This provides
significant security benefits, especially given that what was being tested
was rather small, and could be done in a shell script.

TAMER will receive significant system testing later on.  But for now, none
of this was worth it.

Further audits of dependencies will come later on.  I've always been fairly
insistent on keeping the dependency graph small and auditable, but recent
supply chain attacks have given me a better way to rationalize the security
risk.  Further, I'm the only one on this project right now.
2021-12-02 12:38:06 -05:00
Mike Gerwitz 87c457ba41 tamer: cargo --frozen --offline
Cargo's default behavior is unfortunately to issue network calls each time
it is invoke in order to check for dependencies updates.  This is not only
bad for reproducibility and privacy, but it's also a concern for supply
chain attacks, since most developers are unaware that this is occurring.

Instead, we pin to the lockfile.  Installing dependencies can be done with
`cargo fetch` and updating dependencies must be explicitly done by the
developer, with the lockfile updated.
2021-12-02 11:49:51 -05:00
Mike Gerwitz 54531e2284 tamer: xir::tree::attr: Display impls 2021-11-23 13:05:10 -05:00
Mike Gerwitz ba7ebad930 tamer: obj::xmlo::reader::test: {DUMMY_SPAN=>DS} for brevity
There's a lot of boilerplate that can be reduced in general, but I _really_
want to focus on getting this thing done; I can clean up later.
2021-11-22 11:16:43 -05:00
Mike Gerwitz ba4c32383f tamer: obj::xmlo::reader: Parse root package node attributes
Well, parse to the extent that it was being parsed before, anyway.

The core of this change demonstrates how well TAMER's abstractions work well
together.  (As long as you have an e.g. LSP to help you make sense of all of
the inference, I suppose.)

  Token::Open(QN_LV_PACKAGE | QN_PACKAGE, _) => {
      return Ok(XmloEvent::Package(
          attr_parser_from(&mut self.reader)
              .try_collect_ok()??,
      ));
  }

This finally makes use of `attr_parser_from` and `try_collect_ok`.  All of
the types are inferred---from the iterator transformations, to the error
conversions, to the destination PackageAttrs type.

DEV-10863
2021-11-18 00:59:10 -05:00
Mike Gerwitz d421112f35 tamer: xir::tree::ParserState::store_or_emit: Properly emit Parsed::Done
This was forgotten when the attribute parser was introduced, and led to the
parser continuing to the token following AttrEnd, which properly caused a
failure given that the parser was in the Done state.

There is a future task I have in my backlog to properly address the Done
state, but this is sufficient for now.
2021-11-17 00:13:07 -05:00
Mike Gerwitz e0811589fa tamer: xir::tree::attr::value_atom: Doc typo fix 2021-11-16 15:48:59 -05:00
Mike Gerwitz 7367e20c01 tamer: obj::xmlo: Extract error types into own module 2021-11-16 15:47:52 -05:00
Mike Gerwitz f519dab2b6 tamer: xir::tree::attr::Attr::value_atom: Option<SymbolId>=>SymbolId
To maintain a proper abstraction, this cannot be the responsibility of the
caller; most callers should not know that fragments exist, letalone how to
handle them.
2021-11-16 12:41:03 -05:00
Mike Gerwitz c9be1d613d tamer: iter::collect::TryCollect::try_collect_ok: Doc fix
This was copied from another docblock and I messed it up.
2021-11-16 12:26:05 -05:00
Mike Gerwitz 5233822322 tamer: xir: Remove Text enum
Like previous commits, this replaces the explicit escaping context with the
convention that all values retrieved from `xir` are unescaped on read and
escaped on write.

Comments are a notable TODO, since we must escape only `--`.

CData is also an issue.  I had _expected_ to use it as a means to avoid
unescaping fragments, but I had forgotten that quick_xml hard-codes escaping
on read, so that it can re-use BytesStart!  That is terribly unfortunate,
and may result in us having to re-implement our own read method in the
future to avoid this nonsense.  So I'm just leaving it as a TODO for now.

DEV-11081
2021-11-15 23:47:14 -05:00
Mike Gerwitz 8723ca154d tamer: xir::escape::CachingEscaper: Use new sym::st::ST_COUNT
This adds a constant `ST_COUNT` representing the number of statically
allocated symbols, and uses that to estimate an initial capacity for the
`CachingEscaper`.

This is just a guess (and is certainly too low), but we can adjust later on
after profiling, if it ever comes up.
2021-11-15 21:46:57 -05:00
Mike Gerwitz d710437ee4 tamer: xir::escape::CachingEscaper: New Escaper
As promised, this will cache previously seen escaped/unescaped values by
creating a two-way mapping between them.

DEV-11081
2021-11-15 16:44:24 -05:00
Mike Gerwitz 27ba03b59b tamer: xir::escape: Remove XirString in favor of Escaper
This rewrites a good portion of the previous commit.

Rather than explicitly storing whether a given string has been escaped, we
can instead assume that all SymbolIds leaving or entering XIR are unescaped,
because there is no reason for any other part of the system to deal with
such details of XML documents.

Given that, we need only unescape on read and escape on write.  This is
customary, so why didn't I do that to begin with?

The previous commit outlines the reason, mainly being an optimization for
the echo writer that is upcoming.  However, this solution will end up being
better---it's not implemented yet, but we can have a caching layer, such
that the Escaper records a mapping between escaped and unescaped SymbolIds
to avoid work the next time around.  If we share the Escaper between _all_
readers and the writer, the result is that

  1. Duplicate strings between source files and object files (many of which
     are read by both the linker and compiler) avoid re-unescaping; and
  2. Writers can use this cache to avoid re-escaping when we've already seen
     the escaped variant of the string during read.

The alternative would be a global cache, like the internment system, but I
did not find that to be appropriate here, since this is far less
fundamental and is much easier to compose.

DEV-11081
2021-11-12 14:03:23 -05:00
Mike Gerwitz b1c0783c75 tamer: xir::XirString: WIP implementation (likely going away)
I'm not fond of this implementation, which is why it's not fully
completed.  I wanted to commit this for future reference, and take the
opportunity to explain why I don't like it.

First: this task started as an idea to implement a third variant to
AttrValue and friends that indicates that a value is fixed, in the sense of
a fixed-point function: escaped or unescaped, its value is the same.  This
would allow us to skip wasteful escape/unescape operations.

In doing so, it became obvious that there's no need to leak this information
through the API, and indeed, no part of the system should care.  When we
read XML, it should be unescaped, and when we write, it should be
escaped.  The reason that this didn't quite happen to begin with was an
optimization: I'll be creating an echo writer in place of the current
filesystem-based copy in tamec shortly, and this would allow streaming XIR
directly from the reader to the writer without any unescaping or
re-escaping.

When we unescape, we know the value that it came from, so we could simply
store both symbols---they're 32-bit, so it results in a nicely compressed
64-bit value, so it's essentially cost-free, as long as we accept the
expense of internment.  This is `XirString`.  Then, when we want to escape
or unescape, we first check to see whether a symbol already exists and, if
so, use it.

While this works well for echoing streams, it won't work all that well in
practice: the unescaped SymbolId will be taken and the XirString discarded,
since nothing after XIR should be coupled with it.  Then, when we later
construct a XIR stream for writting, XirString will no longer be available
and our previously known escape is lost, so the writer will have to
re-escape.

Further, if we look at XirString's generic for the XirStringEscaper---it
uses phantom, which hints that maybe it's not in the best place.  Indeed,
I've already acknowledged that only a reader unescapes and only a writer
escapes, and that the rest of the system works with normal (unescaped)
values, so only readers and writers should be part of this process.  I also
already acknowledged that XirString would be lost and only the unescaped
SymbolId would be used.

So what's the point of XirString, then, if it won't be a useful optimization
beyond the temporary echo writer?

Instead, we can take the XirStringWriter and implement two caches on that:
mapping SymbolId from escaped->unescaped and vice-versa.  These can be
simple vectors, since SymbolId is a 32-bit value we will not have much
wasted space for symbols that never get read or written.  We could even
optimize for preinterned symbols using markers, though I'll probably not do
so, and I'll explain why later.

If we do _that_, we get even _better_ optimizations through caching that
_will_ apply in the general case (so, not just for echo), and we're able to
ditch XirString entirely and simply use a SymbolId.  This makes for a much
more friendly API that isn't leaking implementation details, though it
_does_ put an onus on the caller to pass the encoder to both the reader and
the writer, _if_ it wants to take advantage of a cache.  But that burden is
not significant (and is, again, optional if we don't want it).

So, that'll be the next step.
2021-11-10 12:22:10 -05:00
Mike Gerwitz c57aa7fb53 tamer: iter::TryCollect::try_collect_ok: New method
This is intended to alleviate what will be some common boilerplate because
of the Rust compiler error described therein.

This will evolve over time, I'm sure.

DEV-10863
2021-11-10 09:09:07 -05:00
Mike Gerwitz 3140279f04 tamer: iter::trip::TrippableIterator: New trait
This provides convenience methods atop of the already-existing
functions.  These are a bit more ergonomic since they (a) remove a variable
and its generics and (b) are conveniently suggested via LSP (with
e.g. rust-analyzer) if the iterator is of the right type, even if the trait
is not yet imported.  This should help with discoverability as well.
2021-11-05 16:55:46 -04:00
Mike Gerwitz 90e3e94c0a tamer: iter::{TryCollect, TryFromIter}: New traits
These traits augment Rust's built-in traits to handle failure scenarios,
which will allow us to encapsulate lowering logic into discrete,
self-parsing units that enforce e.g. schemas (the example alludes to my
intentions).
2021-11-05 16:33:16 -04:00
Mike Gerwitz 1f01833d30 tamer: xir::tree::attr_parser_from: Do not take ownership over iter
The previous implementation took ownership over the provided iterator, which
was an oversight, considering that this is intended to be used in contexts
where doing so is not possible.  A good example where isolated test cases
aren't necessarily painting the correct picture.

`scan` takes owned values, so this instead uses the same parsing method as
`parse_attrs`, but using a `FromFn` iterator to avoid having to create a
whole new iterator type.  This will work well so long as we don't need to
store the type returned by this (while also wanting to avoid boxing).

DEV-11062
2021-11-05 10:54:05 -04:00
Mike Gerwitz 428d508be4 tamer: {ir::=>}{asg, xir}
See the previous commit.  There is no sense in some common "IR" namespace,
since those IRs should live close to whatever system whose data they
represent.

In the case of these, they are general IRs that can apply to many different
parts of the system.  If that proves to be a false statement, they'll be
moved.

DEV-10863
2021-11-04 16:13:27 -04:00
Mike Gerwitz 5a91db6d54 tamer: obj::xmlo::{legacy=>}ir
Calling it "legacyir" is just confusing.  The original hope, when beginning
TAMER, was that I'd be able to use a new object format in the near future to
help speed up the compilation process.  But that's far from our list of
priorities now, and so seeing "legacy" all over the place is really
confusing considering that it implies that perhaps it shouldn't be used for
new code.

This helps to clear up that cognitive dissonance by remaining neutral on the
topic.  And the reality is that it won't be "legacy" for some time.

DEV-10863
2021-11-04 13:23:38 -04:00
Mike Gerwitz cee6402f8b tamer: Move {ir::legacyir=>obj::xmlo::legacyir}
The IRs really ought to live where they are owned, especially given that
"IR" is so generic that it makes no sense for there to be a single location
for them; they're just data structures coupled with different phases of
compilation.

This will be renamed next commit; see that for details.

This also removes some documentation describing the lowering process,
because it's undergone a number of changes and needs to be accurately
re-summarized in another location.  That will come at a later time after the
work is further along so that I don't have to keep spending the time
rewriting it.

DEV-10863
2021-11-04 13:20:38 -04:00
Mike Gerwitz d06f31b4d3 tamer: obj::xmlo: Compile quickxml even with flag off
This was previous gated behind the negation of the wip-xmlo-xir-reader flag,
which meant that it was not being compiled or picked up by LSP.  Both of
those things are inconvenient and unideal.

DEV-10863
2021-11-04 12:35:08 -04:00
Mike Gerwitz e494f3fdfd tamer: ir::xir::tree::attr_parser_from: New parser iterator
This allows for the lazy parsing of attributes, and makes the necessary
changes to the parser to be able to do so safely without getting into a bad
context.

When XIRT was originally conceived, this concept existed somewhat, but it
was done in a way that would allow the parser to accept invalid input.  This
avoids that problem.

This also introduces the concept of "Done", primarily because we had to for
the AttrEnd token.  This will evolve in following commit(s), which will
allow carrying out the important check of ensuring that the parser has ended
parsing in a valid accepting state (in terms of a state machine).

DEV-11062
2021-11-04 11:04:42 -04:00
Mike Gerwitz 3ba478b09b tamer: ir::xir::tree::ParseError::AttrNameExpected: Display typo fix
We do not want to put backticks around a token display.
2021-11-03 15:07:52 -04:00
Mike Gerwitz adc939d779 tamer: ir::xir::Token: Implement Display
This also modifies xir::tree errors to use Display instead of Debug when
rendering error output.

DEV-10863
2021-11-03 14:54:37 -04:00
Mike Gerwitz c7eb50b636 tamer: xir::xir::tree::parse_attrs: Isolated attribute parsing
This produces an `AttrList` independent from a containing
`Element`.  Upcoming changes may further permit the parser to yield smaller
components that are not part of an aggregate.

DEV-10863
2021-11-03 14:39:03 -04:00
Mike Gerwitz 54e1877d20 tamer: ir::xir::tree: Isolate AttrList parsing
This maintains existing functionality but prepares for an isolated context
for AttrList parsing.

DEV-10863
2021-11-02 14:07:20 -04:00
Mike Gerwitz 6eed728756 tamer: ir::xir::tree: Explicitly list unhandled tokens for exhaustiveness
This allows Rust to carry out its exhaustiveness check for when we add new
tokens.  It further ensure that we understand what we missed, or chose not
to handle.

DEV-10863
2021-11-02 14:07:05 -04:00
Mike Gerwitz edf9a75575 tamer: ir::xir::{QName, Prefix, LocalName}: Implement Display
These will be shown in error messages and need user-friendly
representations.

DEV-10863
2021-11-02 13:55:33 -04:00
Mike Gerwitz d045786cfb tamer: ir::xir::tree::Element::attrs: Wrap in Option
This allows AttrList not only to be lazily initialized (which is less of a
problem at the moment with Vec, but may become one in the future), but also
leaves a space open for attributes to be added _after_ having been
parsed.  It further leaves room to _take_ attributes from their `Element`.

This is important because the next commit will re-introduce the ability to
parse attributes independently, allowing us to put the parser in a state
where we can parse AttrList without an Element context.  To re-use that
parsing under an Element context, we can simply attach an AttrList after it
has been parsed.

Option adds no additional size cost to Vec, so we get this for free (except
for the tiny change that initializes the attribute list when we try to push
to it).

I also think this reads better ("attrs: None").  Though it makes the API
slightly more of a pain to work with.

DEV-10863
2021-10-29 16:34:05 -04:00
Mike Gerwitz a9fd1c7557 tamer: Use TokenStream trait alias where applicable
Simple replacement to improve readability.
2021-10-29 14:39:40 -04:00
Mike Gerwitz 7e6cb2c948 tamer: ir::xir::Token::AttrEnd: New token type
The purpose of this token is to implement a lazy streaming attribute
collection operation without a token of lookup, which would complicate
parsing or require that a TokenStream provide a `peek` method.

This is only required for readers to produce, since readers will be feeding
data to parsers.  I have the writer ignoring it.  If you're looking back at
this commit, the question is whether this was a bad idea: it introduces
inconsistencies into the token stream depending on the context, which can be
confusing and error-prone.

The intent is to have the parser throw an explicit error if the new token is
missing in the context in which it is required, which will safely handle the
issue, but does defer it to runtime.  But only readers need auditing, and
there's only one XIR reader at the moment.

DEV-10863
2021-10-29 13:06:27 -04:00
Mike Gerwitz 18ab032ba0 tamer: Begin XIR-based xmlo reader impl
There isn't a whole lot here, but there is additional work needed in various
places to support upcoming changes and so I want to get this commited to
ease the cognitive burden of what I have thusfar.  And to stop stashing.  We
have a feature flag for a reason.

DEV-10863
2021-10-28 21:21:30 -04:00
Mike Gerwitz ba3b576c93 tamer: ir::xir::qname_const_inner: Fully qualified QName paths
This macro was previously using the path of wherever the template expanded
into, which I found to be unexpected considering that I thought the macros
were hygenic and the names bound to the environment in which they were
defined.

In any case, this solves the problem in all cases.

DEV-10863
2021-10-28 21:19:11 -04:00
Mike Gerwitz f0f58a6e16 tamer: obj::xmlo::asg_builder: Remove example for now
Just until the new xmlo reader is ready, since it will be changing slightly
and fails to compile with the feature flag on now.

DEV-10863
2021-10-28 21:17:53 -04:00
Mike Gerwitz e9871541a8 tamer: benches/iter.rs: Basic benchmark
This was forgotten in the previous commit and exists simply to ensure that
the TripIter doesn't add any significant overhead.  The tests are
a handful of nanoseconds apart, on my machine.
2021-10-28 21:17:41 -04:00
Mike Gerwitz f6c5a224c8 tamer: iter::trip: Introduce initial TripIter concept
See the documentation in this commit for more information.

This is pretty significant, in that it's been a long-standing question for
me how I'd like to join together `Result` iterators without having
unnecessarily complex APIs, and also allow for error recovery.  This solves
both of those problems.

It should be noted, however, that this does not yet explicitly implement
error recovery, beyond being able to observe the failure as the result of
the provided callback function.  Proper recovery will be implemented once
there's a use-case.

DEV-11006
2021-10-28 14:50:41 -04:00
Mike Gerwitz 18cadb9c7d tamer: obj::xmlo::reader: Better organize flagged code
This moves the Iterator impl and From<B> back into `quickxml`.  The type of
the new reader is different, taking an iterator instead of a BufRead.  This
will allow us to easily mock for unit tests, without the clustfuckery that
has ensued previously with quick-xml mocking.

DEV-10863
2021-10-25 13:47:26 -04:00
Mike Gerwitz c76fe87acd tamer: obj::xmlo::reader: Move Xmlo{Result,Error,Event}
These will need an API change, but are otherwise shared.  This means that
only the XmloReader is gated.
2021-10-25 12:26:25 -04:00
Mike Gerwitz f7d8aa1e4f tamer: wip-xml-xir-reader flag and setup
The original plan was to modify the existing reader to use the new
XmlXirReader, but that's going to be a lot of ongoing uncommitted work, with
both tests and implementation.  The better option seems to be to reimplement
it, since so many things are changing.

This flag will be short-lived and removed as soon as the implementation is
complete.

DEV-10863
2021-10-25 12:02:46 -04:00
Mike Gerwitz e6f53c20fd tamer: ir::xir::reader: Disable quick-xml check_end_names
XIR must support tag mismatches; XIRT will validate them.

This is currently disabled in the linker's xmlo reader as well.

DEV-10863
2021-10-25 10:58:19 -04:00
Mike Gerwitz d72ab3675c tamer: ir::xir::reader: Comment parsing
Comments re-use Text, but they are _not_ escaped, so we need to take care
with the type to ensure that, if the value were ever used with a
Token::Text, that we don't end up injecting XML.
2021-10-21 22:04:45 -04:00
Mike Gerwitz fdb8e5998c tamer: ir::xir::reader: CData parsing
quick_xml provides us the value escaped, so we can just handle this the same
way as Text for now.

In the future, we may want to distinguish between the two so that we can
reconstruct an identical XML document, but at the moment CData isn't used at
all in TAME sources or outputs, and so I'm not going to worry about it for
now.

DEV-10863
2021-10-21 21:55:15 -04:00
Mike Gerwitz 8b212959c8 tamer: ir::xir::reader: Text and mixed content
It's nice being able to breeze through changes, since that's been a pretty
rare thing so far, given all the foundational work that has been needed.

This should get us pretty damn close to being able to parse the `xmlo` files
for the reader linker, if we're not there already.

DEV-10863
2021-10-21 21:44:04 -04:00
Mike Gerwitz 13a779ec9c tamer: ir::xir::reader: Remove namespace TODO
This isn't XIR's responsibility, and so there's nothing to do here.
2021-10-21 16:52:58 -04:00
Mike Gerwitz 6d25be0ec7 tamer: ir::xir::reader: Refactor common element open parsing
As mentioned in the previous commit, this is just minor cleanup.
2021-10-21 16:51:47 -04:00
Mike Gerwitz e18aeeffac tamer: ir::xir::reader: Parsing of child nodes
This is quick-and-dirty; refactoring can be done later on.  This is also
intended to demonstrate the ease with which additional events can be
added---the hard work is done.
2021-10-21 16:32:19 -04:00
Mike Gerwitz 4c4d89f84f tamer: ir::xir::reader: Initial concept
This is an initial working concept for the reader which handles, so far,
just a single attribute.  But extending it to completion will not be all
that much more work.

This does not have namespace support---that will be added later as part of
XIRT, which is responsible for semantic analysis.  This allows XIR to stay
wonderfully simple, and won't have any impact on the writer (which expects
that QNames are unresolved and contain the namespace prefix to be written).
2021-10-21 16:23:11 -04:00
Mike Gerwitz fc3953e90e tamer: benches/sym.rs: Interner::intern_utf8 benchmarks
These were forgotten in the previous commit.
2021-10-19 13:42:26 -04:00
Mike Gerwitz b8d0da9095 tamer: sym::Interner::intern_utf8
This is the safe version of the existing intern_utf8_unchecked, and exists
as a performance optimization.

We're about to introduce a XIR reader, which is going to intern a _lot_ of
duplicate strings, since it will intern node and attribute names as
well.  Given that, we do not want to spent a lot of time performing UTF-8
checks that have already been performed.

We know that, if an intern is in the pool, it's either already UTF-8 or that
check was bypassed when it was initially interned.  Therefore, if we find an
existing symbol, that can be returned without having to perform any
check.  Otherwise, we intern as we usually would after attempting to convert
the byte slice into a string.

This allows us to continue to have good performance for interning without
sacrificing safety for strings.
2021-10-19 12:56:57 -04:00
Mike Gerwitz 63e5a0d441 tamer: benches/sym.rs: Add additional UTF-8-related tests
The intent of this is to demonstrate how significant of an impact checking
byte arrays for UTF-8 validity will have, since the existing tests do not
make that clear (a static string in Rust is always valid UTF-8).

These benchmarks show that the cost when re-interning an already existing
value is +50%.

This is important, because the new reader will be interning a _lot_ of
duplicate strings, whereas the existing reader operates on byte arrays
without interning unless necessary.  And, when it does, it does so
unchecked.  But we'd rather not do that, since we cannot guarantee that
those XML files are valid (and not modified in some way).

Upcoming commits will have what I think is a reasonable compromise to this,
based on the fact that we'll be encountering _many_ duplicate strings in
parsing XML files.

DEV-10920
2021-10-18 21:35:32 -04:00
Mike Gerwitz 2715f3e845 tamer: sym: Expose raw SymbolId for static symbols
This provides a child `raw` module that exposes a SymbolId representing the
inner value of each of the static newtypes.  This is needed in situations
where the type must match and the type of the static symbol is not
important.

In particular, when comparing against runtime-allocated symbols in `match`
expressions.

It is also worth noting that this commit managed to hit a bug in Rustc that
was fixed on 10/1/2021.  We use nightly, and it doesn't seem that this
occurred in stable, from bug reports.

  - https://github.com/rust-lang/rust/issues/89393
  - 5ab1245303
  - Original issue: https://github.com/rust-lang/rust/issues/72476

The error was:

  compiler/rustc_mir_build/src/thir/pattern/deconstruct_pat.rs:1191:22:
  Unexpected type for `Single` constructor: <u32 as sym::symbol::SymbolIndexSize>::NonZero

  thread 'rustc' panicked at 'Box<dyn Any>', compiler/rustc_errors/src/lib.rs:1146:9

This occurred because we were trying to use `SymbolId` as the type, which
uses a projected type as its inner value: `SymbolId<Ix: SymbolIndexSize>(Ix::NonZero)`.
This was not a problem with the static newtypes because their inner type was
simply `SymbolId<Ix>`, which is not projected.

This is one of the risks of using nightly.

But, the point is: if you receive this error, upgrade your toolchain.
2021-10-18 10:53:53 -04:00
Mike Gerwitz 581b9d4e65 tamer: Use `..` for tuple unimportant variant matches
Tbh, I was unaware that this was supported by tuple variants until reading
over the Rustc source code for something.  (Which I had previously read, but
I must have missed it.)

This is more proper, in the sense that in a lot of cases we not only care
about how many values a tuple has, but if we explicitly match on them using
`_`, then any time we modify the number of values, it would _break_ any code
doing so.  Using this method, we improve maintainability by not causing
breakages under those circumstances.

But, consequently, it's important that we use this only when we _really_
don't care and don't want to be notified by the compiler.

I did not use `..` as a prefix, even where supported, because the intent is
to append additional information to tuples.  Consequently, I also used `..`
in places where no additional fields currently exist, since they may in the
future (e.g. introducing `Span` for `IdentObject`).
2021-10-15 12:28:59 -04:00
Mike Gerwitz 739cf7e6eb tamer: ir::asg::object::IdentObject: Define methods from IdentObjectData
In particular, `name` needn't return an `Option`.  `fragment` also returns a
copy, since it's just a `SymbolId`.  (It really ought to be a newtype rather
than an alias, but we'll worry about that some other time.)

These changes allow us to remove some runtime panics.

DEV-10859
2021-10-14 14:38:02 -04:00
Mike Gerwitz f055cb77c2 tamer: ld::xmle: Narrow Sections types
This moves the logic that sorts identifiers into sections into Sections
itself, and introduces XmleSections to allow for mocking for testing.

This then allows us to narrow the types significantly, eliminating some
runtime checks.  The types can be narrowed further, but I'll be limiting the
work I'll be doing now; this'll be inevitably addressed as we use the ASG
for the compiler.

This also handles moving Sections tests, which was a TODO from the previous
commit.

DEV-10859
2021-10-14 12:40:13 -04:00
Mike Gerwitz ea11cf1416 tamer: ld::xmle::lower: Extract sectioning into Sections
This is the appropriate place to be, now that we've begun narrowing the
types.  We'll be able to do so further; this is just the first step.

This does not yet move the tests, but the code is still tested because it's
tightly coupled with `sort`.  Those will move in the next commit(s).

DEV-10859
2021-10-12 12:15:11 -04:00
Mike Gerwitz 08d92ca663 tamer: ld::xmle::sections: Remove generic object type
xmle sections will only ever contain an object of one type, so there is no
use in making this generic.

I think the original plan was to have this represent, generically, sections
of some object file (like ELF), but doing so would require a significant
redesign anyway, so it makes no sense.  This is easier to reason about.

DEV-10859
2021-10-12 10:35:14 -04:00
Mike Gerwitz 31144d0c9a tamer: benches/asg_lower.rs: Add missing file from previous commit
This was missed in the `lower` module move.
2021-10-12 10:30:35 -04:00
Mike Gerwitz 27480229df tamer: ld (Linking Process): Minor doc update to reflect changes
DEV-10859
2021-10-12 09:49:40 -04:00
Mike Gerwitz df328da71f tamer: ir::asg::SortableAsg: Move into ld::xmle::lower
This has always been a lowering operation, but it was not phrased in terms
of it, which made the process a bit more confusing to understand.

The implementation hasn't changed, but this is an incremental refactoring
and so exposes BaseAsg and its `graph` field temporarily.

DEV-10859
2021-10-12 09:49:33 -04:00
Mike Gerwitz 81ec65742a tamer: {ir::asg=>ld::xmle}::section
Sections, as written, are specific to xmle files.

I think the intent originally was to have this be more generic, but that
doesn't really make sense.

By explicitly coupling it with `xmle` files, that will allow us to turn this
into a proper lowering operation with its own validations that will allow
`xmle::xir` to do its job without having to validate anything itself.
2021-10-12 00:05:44 -04:00
Mike Gerwitz 1c181b568d tamer: ld::poc: Update comment reflecting current state
The linker is feature-complete, but this file has lived on because the
project was on pause for quite some time.
2021-10-11 23:54:24 -04:00
Mike Gerwitz f899ac898e tamer: {obj=>ld}::xmle
This is a linker-specific module.
2021-10-11 23:52:59 -04:00
Mike Gerwitz 5ea5cffd09 tamer: relroot String->SymbolId
This was [one of] the last remaining Strings; SymbolId should be used across
the board.
2021-10-11 16:00:19 -04:00
Mike Gerwitz 7873d46afb tamer: Replace all &'static str in errors with SymbolId
Now that SymbolId implements Display and resolves, this works out well.
2021-10-11 15:39:53 -04:00
Mike Gerwitz 7e9271e189 tamer: span: Primitive Display impl
This outputs enough information to be a little bit useful in the event of an
error.  In the future, we'll want to provide a (likely non-Display)
implementation that provides line number and source file context with
the problem characters indicated, like Rust.
2021-10-11 14:14:43 -04:00
Mike Gerwitz a9140730d9 tamer: sym: Implement Display for SymbolId
This is a significant departure from my original plans---this makes it
_easy_ to display symbol values, despite me not wanting that to occur unless
absolutely necessary.

The reality is, based on the design of the system, they will only occur in
these situations:

  1. Writing to files;
  2. Displaying errors;
  3. Tests; or
  4. People not following the design of the system.

The fourth one is the most risky as people begin to contribute in the
future, but the reality is that those can be fixed as they are encountered,
since if they're not showing up in a profiler, then they must not be causing
much of a problem.
2021-10-11 13:52:35 -04:00
Mike Gerwitz 85909f1590 tamer: sym::SymbolStr: Remove
This removes `SymbolStr` in favor of, simply, `&'static str`.

The abstraction provided no additional safety since the slice was trivially
extracted (and commonly, in practice), and was inconvenient to work with.

This is part of a process of relaxing lookups so that symbols can be
conveniently displayed in errors; rather than trying to prevent the
developer from doing something bad, we'll just rely on conventions, hope
that it doesn't happen, and if it does, address it either at that time or
when it shows up in the profiler.
2021-10-11 12:58:48 -04:00
Mike Gerwitz 68397f1413 tamer: ir::xir: Add missing docs for QName, Prefix, LocalName
The docs still need to be improved, but they can be touched as we go.

This concludes the initial development of XIR.  That was much more involved
that I had originally intended, but the result is good.

DEV-10561
2021-10-11 11:56:03 -04:00
Mike Gerwitz bc5091d2a7 tamer: ir::xir (newtype_symbol!): Remove for now
This does not belong here and was more of a POC at the time.  It can be
added later on when I have the time; I have to move on.
2021-10-11 11:51:51 -04:00
Mike Gerwitz f65ec818ab tamer: obj::xmle::xir: Correct doc typos Xml{e=>}Writer 2021-10-11 11:51:32 -04:00
Mike Gerwitz 3e385d1a1b tamer: obj::xmle::xir: Finalize docs
This could be improved upon, but there will be more work coming up for this
to finalize Sections.

DEV-10561
2021-10-11 11:43:49 -04:00
Mike Gerwitz bc5e8ebe75 tamer: obj::xmle::xir: Extract ElemWrap into ir::xir::iter
This generalizes it a bit and provides tests, which was always the intent;
the existing code was POC to determine if this could be done without
performance degradation (see that commit for more information).
2021-10-11 10:33:24 -04:00
Mike Gerwitz cde08b125c tamer: span (DUMMY_SPAN): New constant
Rather than having to use lazy_static! in all these tests, we can derive an
unlimited number of dummy spans from this one using e.g. `offset_add`.
2021-10-11 10:29:58 -04:00
Mike Gerwitz cf239531e0 tamer: span (offset_add): New method
More will come in the future, including the ability to add two spans.
2021-10-11 10:28:47 -04:00
Mike Gerwitz de3d7ef393 tamer: span: Introduce twospan
The intent is to support the composition and decomposition of spans such
that (A, B) is as documented here.  This only performs the trivial case for
the sake of providing a convenient API when the developer would otherwise
just type (S, S).
2021-10-11 09:56:48 -04:00
Mike Gerwitz 1a2f6bd209 tamer: obj::xmle::xir: Extract ElemWrap into ir::xir::iter 2021-10-11 09:34:17 -04:00
Mike Gerwitz de62a2acbc tamer: ir::asg::section: Reduce fields
This is intended to represent the sections written to the final xmle file,
and there was unnecessary complexity in separating everything.

By reducing this IR further, we can begin to constrain its types to
eliminate some of the runtime panics and error checking we have/had in the
writer.
2021-10-11 09:07:48 -04:00
Mike Gerwitz f70f5653b2 tamer: ir::asg::section: Head and tail can have only one object
This is the beginning of a refactoring to simplify this implementation a
little bit.
2021-10-09 00:27:03 -04:00
Mike Gerwitz 0626629cb3 tamer: Remove old xmle writer and wip-xir-xmle-writer flag
The new writer has reached parity of the old, with the exception of some
edge case explicit error handling that should never occur (which will be
added), and cleanup/docs.

Removing this flag now allows me to perform that cleanup without having to
worry about updating the now-old implementation.

I ran `tameld` with the new writer against our production system with
numerous programs and a significant number of test cases, and diff'd the old
and new xmle files, and everything looks good.
2021-10-08 22:04:42 -04:00
Mike Gerwitz 82727a5d66 tamer: obj::xmle::xir::header: Remove Rust 2018 comment
We're on 2021 now.
2021-10-08 21:43:28 -04:00
Mike Gerwitz d616d9475c tamer: obj::xmle::xir: Complete writer functionality
This is a significant milestone, in the sense that it is the culmination of
the past month or so of work to prove that an Iterator-based XIR will be
viable for the system.

This barely had any impact on the performance from the previous commit
reporting the profiling.  This performs at least as well as the quick-xml
based writer.  In isolated benchmarks, it performs better, but in the real
world, the linker spends most of its time reading xmlo files, and so minor
differences in writing do not have a significant overall impact.

With that said, a lot of cleanup and documentation is still needed.  That is
the subject of the upcoming commits, before this writer can finalized.
2021-10-08 16:37:46 -04:00
Mike Gerwitz 929a6c9815 tamer: obj::xmle::xir::tree: Parse Text into Element
This simply adds support for Text nodes as a child of Element.  This support
unit tests for the upcoming change for xmle fragments.
2021-10-08 16:16:33 -04:00
Mike Gerwitz f0f6f89745 tamer: Makefile.am (bench-build): New target, default for all
Build the benchmarks by default to catch breakages without having to incur
the cost of actually running them.
2021-10-08 09:27:56 -04:00
Mike Gerwitz 75d2ecf4dd tamer: obj::xmle::xir: Consideration of simplified iterators
The previous iterators had to be used in a certain order because they mixed
concerns, out of concern for performance.  This attempts to chain even more
iterators to see how it may perform.

To be clear: this will be cleaned up.  This was just an experiment.

Here were profiles on the average of 50 runs of linking our largest program:

  Baseline, pre-XIR (with fragments removed from output)               0.8082
  XIR writer, pre-ElemWrap, no #[inline]                               0.7844s
  XIR writer, ElemWrap, no #[inline]                                   0.7918s
  XIR writer, ElemWrap, inlines in obj::xmle::xir                      0.7892s
  XIR writer, ElemWrap, inlines in obj::xmle::xir and ir::asg::section 0.7858s
  XIR writer, ElemWrap, inline in only ir::asg::section                0.781s
  Pre-ElemWrap, inlines in ir::asg::section                            0.7772s

These profiles are difficult, because they hit the filesystem so much.  I
write to /dev/null, but it reads 100s of xmlo files from disk.

It's clear that the impact is fairly modest and within a margin of error; as
such, I will continue down the path of writing code that's easier to grok
and maintain, since not doing so would be a micro-optimization relative to
the concerns of the rest of the system at this point.

But the purpose of all of this work was to determine whether an
iterator-based XIR would be viable.  It seems to be competitive.  I'll
finish up the writer reimplementation and move on.
2021-10-07 16:48:58 -04:00
Mike Gerwitz 7f5064c665 tamer: obj::xmle::xir: Write l:map-from
This contains some awkward coupling for opening and closing tags to reduce
the complexity of the `Iterator` types that must be manually
specified.  That may be addressed shortly.
2021-10-05 16:13:47 -04:00
Austin Schaffer d54ef62a0d Fix import ordering 2021-10-04 17:15:02 -04:00
Mike Gerwitz 1a44e04333 tamer: ld: Write is unused outside of flag 2021-10-04 16:34:25 -04:00
Mike Gerwitz e2c9944f1b tamer: Move Sections map from unique from writer into Sections
We're implementing an new XIR-based writer and don't want to have to
duplicate this; it didn't really belong there to begin with.
2021-10-04 16:31:30 -04:00
Mike Gerwitz 004f5dc312 tamer: Read only a single map preproc:from from xmlo files
This was creating a heap-allocated `Vec` for each map symbol despite not
actually needing it.  We do have multiple froms for return map values.

But by the time we may want this type of thing, we'll have a different IR
for it anyway.
2021-10-04 14:59:33 -04:00
Mike Gerwitz 772619f6f0 tamer: Replace explicit array::IntoIter::new with IntoIter
Now that we're on 2021 Edition, the default behavior has changed to be
consistent.
2021-10-02 01:03:19 -04:00
Mike Gerwitz f9c9c95516 tamer: sym::prefill: Static symbol polymorphism
See the docs for a much deeper discussion.  In summary: traits do not
support static methods, and this is the workaround, which relies on unstable
nightly constant function features.

This implementation is tested using `qname_const!`, and will be utilized
with a new static type in a following commit.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 9d87962e96 tamer: Use Rust 2021 Edition
This will be stable Oct 21; this uses nightly for now.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 885d5e4d8f tamer: Switch back to nightly toolchain
This is to support two things:
  1. Early switch to 2021 Edition, which is stable Oct 21; and
  2. To make use of unstable const features.

The rationale is that switching to nightly does not really have any
significant downside for us, given that TAMER is used only by us and
the only risk is that unstable features may change a bit, which can be
mitigated with certain precautions.

The rationale for each unstable feature will be documented as they are used,
including documentation on what would be required to remove it and what
functionality would be lost / need to change in doing so.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 7c61a92d30 tamer: obj::xmle::xir: Minor clean and docs
This is far from fully documented; it's just a start.  I'll document fully
once the implementation is done, to ensure I don't waste time documenting
things that may change.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 42188e80e7 tamer: obj::xmle::xir::test: Extract into own file
These are getting large and messy.

And I now notice that I never completed the header test after
prototyping.  Shame on me.

Also, errata from the previous commit message: the diffs are identical
_except for attribute escaping_ that is unnecessary; we're outputting data
read directly from existing XML files (output by Saxon), so characters are
already escaped as needed.

DEV-10561
2021-10-02 00:58:13 -04:00
Mike Gerwitz 7269e68b00 tamer: obj::xmle::xir: Complete l:dep
The `l:dep` section of the `xmle` file, after formatting (since XIR writes
without newlines and indentation), is now identical to the existing xmle
writer.  I can now move on to the other sections.

Note that the attribute movement in this commit is simply to get the diff to
properly align.  Once the current xmle writer is removed, I'll organize them
a bit more sensibly.

`obj::xmle::xir` also needs documentation, now that it's shown to be viable.
2021-09-30 13:06:30 -04:00
Mike Gerwitz acf55fad81 tamer: Intern desc from xmle on read
The new xmle writer was having to intern before write, which did not make
sense.

This continues with consistently using symbols throughout the system, and
is a smaller size than `String` as a bonus.
2021-09-29 23:31:07 -04:00
Mike Gerwitz 5250571f15 tamer: ir::asg::ident: Use symbols in place of string slice mapping
`IdentKind` needs to be written to `xmle` files and displayed in error
messages.  String slices were used when quick-xml was used for writing,
which will be going away with the new writer.
2021-09-29 23:18:23 -04:00
Mike Gerwitz fa4181770f tamer: src::ir::asg::ident::Dim: Assert n<10
This replaces a TODO with an assertion.
2021-09-29 16:26:41 -04:00
Mike Gerwitz 6864fbc1cd tamer: Start of XIR-based xmle writer
This has been a long time coming, and has been repeatedly stashed as other
parts of the system have evolved to support it.  The introduction of the XIR
tree was to write tests for this (which are sloppy atm).

This currently writes out the `xmle` header and _most_ of the `l:dep`
section; it's missing the object-type-specific attributes.  There is,
relatively speaking, not much more work to do here.

The feature flag `wip-xir-xmle-writer` was introduced to toggle this system
in place of `XmleWriter`.  Initial benchmarks show that it will be
competitive with the quick-xml-based writer, but remember that is not the
goal: the purpose of this is to test XIR in a production system before we
continue to implement it for a frontend, and to refactor so that we do not
have multiple implementations writing XML files (once we echo the source XML
files).

I'm excited to get this done with so that I can move on.  This has been
rather exhausting.
2021-09-28 14:52:53 -04:00
Mike Gerwitz 863d990cbd tamer: sym: 16-bit static symbol prefill
The 16-bit interner at present will be used only for span contexts.  In the
future, this interner may become specialized specifically for that, but for
now let's just re-use what we already have so that I can move on.

DEV-10733
2021-09-28 10:39:46 -04:00
Mike Gerwitz 96b16c6de9 tamer: sym::prefill::test::global_sanity_check: Note duplicate strings
I want to make it clear in the assertion that the problem could be caused by
duplicate strings.  We do not sort by string, because in part we may in the
future want to group certain symbols together in some arbitrary way so we
can compare ranges (using the markers).

If that doesn't end up happening, it may be better to just sort by string
to obviate the problem.
2021-09-24 16:25:29 -04:00
Mike Gerwitz db8a098452 tamer: sym: Minor documentation refinement
Mostly rewording.
2021-09-24 10:11:19 -04:00
Mike Gerwitz c71d36b154 tamer: sym::prefill: All-caps constants for static symbols
It's really awkward not having them caps, when not only are constants
expected to be, but also that we cannot maintain consistency between the
string and the identifier name in even the simplest of cases.

(We could use `r#`, but that's too cumbersome.)
2021-09-23 23:48:28 -04:00
Mike Gerwitz 785ca0fe9e tamer: sym::prefill: Remove StaticSymbolId in favor of refined types
`StaticSymbolId` was created before the more specific types, which render it
unnecessary.  If we need a generic type, it can be re-introduced, but using
`static_symbol_newtypes!`.
2021-09-23 23:35:45 -04:00
Mike Gerwitz 15ff00b3cf tamer: sym: Only prefill 32-bit global interner
This is the interner that is intended to be used with the majority of the
system; the 16-bit interner is left around for the moment, but will likely
later become specialized.
2021-09-23 16:11:17 -04:00
Mike Gerwitz e91aeef478 tamer: Remove Ix generalization throughout system
This had the writing on the wall all the same as the `'i` interner lifetime
that came before it.  It was too much of a maintenance burden trying to
accommodate both 16-bit and 32-bit symbols generically.

There is a situation where we do still want 16-bit symbols---the
`Span`.  Therefore, I have left generic support for symbol sizes, as well as
the different global interners, but `SymbolId` now defaults to 32-bit, as
does `Asg`.  Further, the size parameter has been removed from the rest of
the code, with the exception of `Span`.

This cleans things up quite a bit, and is much nicer to work with.  If we
want 16-bit symbols in the future for packing to increase CPU cache
performance, we can handle that situation then in that specific case; it's a
premature optimization that's not at all worth the effort here.
2021-09-23 14:52:54 -04:00
Mike Gerwitz ed245bb099 tamer: sym::prefill: Initial typed static symbol concept
We'll see how the syntax evolves over time.  It's not ideal to have to
specify the type, rather than having the compiler infer it, but I don't much
feel like getting into my first procedural macro right now, so we'll stick
with this approach for the time being.

This will set the stage to be able to safely e.g. create QNames statically
at compile-time and would allow us to make any attempts to bypass it
unsafe.
2021-09-23 00:37:39 -04:00
Mike Gerwitz b972b0b202 tamer: sym::StaticSymbolId: Introduce
Previously, we were allocating only u32 versions of `SymbolId` for the
statically allocated symbols.  This introduces a new symbol type with a very
small datatype (8 bits) that is able to cast into any `SymbolId`.  This is
explained in the docs.

We'll be taking this typing further in future commits so that static symbols
are better-suited for compile-time guarantees for static newtype
construction.

DEV-10710
2021-09-22 21:37:06 -04:00
Mike Gerwitz c87147c277 configure.ac: Bump Rust 1.{53=>54} for using macros in attribute values
The previous commit uses `concat!` for doc generation.  I forgot that this
was only recently stabalized.
2021-09-22 16:47:17 -04:00
Mike Gerwitz 366fef714b tamer: sym::prefill: Introduce static symbols
This is the beginning of static symbols, which is becoming increasing
necessary as it's quite a pain to have to deal with interning static strings
any place they're used.

It's _more_ of a pain to do that in conjunction with newtypes (e.g. `QName`,
`AttValue`, etc) that make use of `SymbolId`; this will allow us to
construct _those_ statically as well, and additional work to support that
will be coming up.

DEV-10701
2021-09-22 16:08:40 -04:00
Mike Gerwitz e0a209d417 tamer: bench: xir: Reduce writer benchmark memory usage
These were using GiB of memory, which is ...unnecessary.

I reduced the iteration count significantly, but it was still wasting a lot
of time and memory and needed `with_capacity` to reduce the number of copies
after reallocation.

It is not typical that a buffer would contain this much information.
2021-09-21 16:21:32 -04:00
Mike Gerwitz aee781a6fb tamer: bench: xir: Fix broken benchmark
This broke when I removed `SelfClose`.  I used to run
`make all fmt check bench` before every push, but they take a while to run,
in part because it uses nightly and has to recompile too.

But it looks like I need to be more diligent again.
2021-09-21 16:09:50 -04:00
Mike Gerwitz b348892276 tamer: ir::xir::tree: Introduce attribute fragment parsing
This is exactly was I said I was _not_ going to do in the previous commit,
but apparently hacking late at night had me forget the whole reason that
XIRT is being introduced now---unit tests.  I'll be emitting a XIR stream
and I need to parse it for convenience in the tests.

So, here's a good start.  Next will be some generalizations that are useful
for the tests as well.  This is pretty bare, but accomplishes the task.

See docs for more info.
2021-09-21 16:07:38 -04:00
Mike Gerwitz a5afc76568 tamer: ir::xir::tree: Extract Attr{,List} into new module
The `tree` module is getting more difficult to navigate.  The tests still
remain where they were, since a bunch of concerns are mixed together.  Any
tests specific only to this module will be added here.
2021-09-21 10:43:23 -04:00
Mike Gerwitz fe7b64fe62 tamer: ir::xir::tree::AttrName: Remove unused, rename {Ele=>}AttrName
Attributes used to be able to be emitted standalone, but that was abandoned
a while back to clean things up a bit.  This cleanup was missed.
2021-09-21 09:29:56 -04:00
Mike Gerwitz c6a7988bc8 tamer: ir::xir: Add Token::AttrValueFragment with writer support
This is implemented only for the writer, since its use case is to be able to
concatenate strings without copying during writing.

It doesn't really make sense to support this in XIR Tree, since a reader
should never produce this.  But if we ever run into this (e.g. due to some
internal processing pipeline), we'll address it then; XIR Tree might have to
do copying, then, but should probably wait until encountering all fragments
before interning.  That'd be a distraction right now.
2021-09-21 00:16:30 -04:00
Mike Gerwitz e95afe2658 tamer: ir::xir::tree::Element::open: Fix doc typo 2021-09-21 00:16:30 -04:00
Mike Gerwitz 3bb6f0cf35 tamer: ir::asg::ident: AsRef impls for SymbolId types
This commit will make more sense once the broader context is committed, but
it's needed for lowering from `Sections` into a XIR stream.

This will also change once we pre-allocate symbols, like rustc, when the
interner is initialized.

This is my first use of the `paste` crate, which is used to generate
identifiers.  So this is partly an experiment, and it seems much better than
having to write a proc macro, at least at this point in time.  If this code
stays around, it'll probably be generalized further and used elsewhere, but
I'd prefer not to go this route long-term.
2021-09-20 16:50:40 -04:00
Mike Gerwitz 12daddcc2d tamer: ir::xir::tree::Element: Open element constructor
This simply moves the construction into `Element`.
2021-09-16 10:52:00 -04:00
Mike Gerwitz ea50e1112a tamer: ir::xir::tree: Extract tests into own file
This file's getting large, and will only grow more complex.
2021-09-16 10:18:02 -04:00
Mike Gerwitz 3484336b1d tamer: ir::xir::tree::Stack: Encapsulate ElementStack manipulation
This moves some logic into `ElementStack` (which would be part of `Stack` if
variants were their own types), rather than peering so deeply into its
data.
2021-09-16 10:07:37 -04:00
Mike Gerwitz a49ac23aeb tamer: ir::xir::tree: Child element attribute parsing
This correctly retains and restores the parent stack after processing an
attribute for a child element.

This does increase the size of [`Stack`] a bit, but we can evaluate whether
it's too large at a later time.  It's currently 832 bits with `Ix=u32`,
which is large, but the question is whether it matters; we'll see as we
begin to use it.
2021-09-15 16:46:15 -04:00
Mike Gerwitz 61e493066c tamer: ir::xir::tree: Clean up parser implementation
This moves most of the parsing logic into `Stack`, which rightfully owns the
stack manipulation and state transitions.  `ParserState` becomes exactly
what it says it is---a management of the persistent state of the parser, and
is also responsible for digesting tokens and dispatching their data to the
proper event.

This approach has a number of benefits over the old design: it's
self-documenting, making the intent clear; and it is easier to reason about
the subset of states (for both humans and Rusts) than a large match of
transitions.

This contains a number of TODO items that will be addressed shortly.  It
also obviated that the previous commit was incomplete---it doesn't persist
`pstack` for attributes on child elements!  That'll be fixed too.
2021-09-15 16:33:08 -04:00
Mike Gerwitz 366ecca8ea tamer: ir::xir::tree: Initial child element parsing
This modifies the tree parser to handle child elements.  It's mostly
proof-of-concept code; the next commit will clean it up a bit so that it's
largely self-documenting.
2021-09-15 11:19:08 -04:00
Mike Gerwitz 51507ccdad tamer: ir::xir: Combine Token::{SelfClose, Close} variants
This removes `SelfClose` and merges it with `Close` by making the first
parameter an `Option`.  This isn't really ideal, but it really simplifies
pattern matching, especially for the next commit.  I'll have more details
there.

The primary motivation was lack of stabalization for binding after `@` in
matches, e.g. `Foo(name, ele) | ele @ Element { name, .. }`.  It looks like
it's ready, though; maybe next Rust release?

  https://github.com/rust-lang/rust/issues/65490

I don't know if I'll revert this change after then.  This seems plenty
clear, albeit more verbose.
2021-09-13 13:06:20 -04:00
Mike Gerwitz 1c40b9c504 tamer: ir::xir::tree: Closing element parsing with balance check
This introduces parser errors, but does not yet support error recovery; that
problem will be discussed in a commit in the near future, after the writer
is sorted out a bit more.

DEV-10561
2021-09-13 10:45:38 -04:00
Mike Gerwitz 5979e1fb90 tamer: ir::xir::tree: Correct italic formatting in docs
I was using an Org mode format.
2021-09-13 09:47:39 -04:00
Mike Gerwitz fd8a05164d tamer: ir::xir::tree: Remove Tree::Attr, add AttrList
The idea, previously, was that parsing could begin at attributes selectively
and be parsed independently.  But that's really awkward with `Tree`, since
it effectively allows orphan attributes as children of an
`Element`.  Nonsense.

Instead, if we truly only want an attribute list, we can offer a function to
create a parser with an empty `Stack::BuddingElement` that can accumulate
them.
2021-09-09 14:40:58 -04:00
Mike Gerwitz 4987bc39b0 tamer: ir::xir::tree::parser_from: Yield parsed trees
Previously, `parser_from` was a simple wrapper around `parse`; now, this
provides a more convenient API where `next` will yield the next parsed
object.

See docs for much more information and rationale.
2021-09-09 13:05:11 -04:00
Mike Gerwitz 1452a4186a tamer: convert: Add missing method-level docs 2021-09-08 16:12:53 -04:00
Mike Gerwitz 2586827d64 tamer: convert::{ExpectFrom, ExpectInto}: New traits
These traits are intended to eliminate boilerplate, primarily in tests, in
situations where from/into is not expected to fail.

Given that TAMER must only panic for internal compiler errors, this should
not often be used outside of test cases.  Further, there may be better
options in the future (e.g. QNames could be statically compiled rather than
trying to convert at runtime, in this case).
2021-09-08 16:03:44 -04:00
Mike Gerwitz 12bb88e4b5 tamer: ir::xir::tree: Introduce XIR tree
This begins to introduce the XIR tree.  I was originally going to wait on
this until after implementing the xmle writer in terms of XIR, but writing
unit tests is too much of a pain on the stream, so now is as good of a time
as any.

This has very limited support so far; it'll be added to as time goes on.
2021-09-08 13:56:04 -04:00
Mike Gerwitz ab093046e9 tamer: ir::asg::section: Provide iterators for major section groups
These groups happen to correspond with the sections of the xmle file, which
suggests again that this lives in the wrong place.  But I should really have
my focus elsewhere right now, so I don't know if I'll go any further right
now.  I guess we'll see as the writer is reimplemented.
2021-09-01 11:21:44 -04:00
Mike Gerwitz 1fa9614698 tamer: ir::asg::section: Improve iteration
`SectionsIter` was introduced to remove that responsibility from xmle
writer, since that's currently being reimplemented using XIR.

The existing iterator has been renamed SectionIter{ator=>} for a more
idiomatic name for iterator structs, and now has a static type rather than
relying on dynamic dispatch.  The author of that code wasn't sure how to
handle it otherwise.  (Which is understandable, since we were both still
getting acquainted with Rust.)  There's no notable change in performance in
my benchmarking.

This abstraction is a bit awkward, in that it's named for object file
sections, but they aren't.  Further, it's coupled with the ASG via
`SortableAsg` and perhaps should be generalized into a sorting routine that
takes a function for sorting, so that `Sections` can be moved into xmle's
packages.
2021-09-01 09:14:51 -04:00
Mike Gerwitz b80064f59e tamer: configure: Check for Rust 1.{52=>53}.
Or-pattern syntax is used; I had forgotten to bump this version.

For example, match on `Foo(Bar | Baz)` vs. `Foo(Bar) | Foo(Baz)`.
2021-08-30 15:19:14 -04:00
Mike Gerwitz 0a8fb71c1b tamer: tameld: Use buffered writes
This was an oversight.  The difference is significant.  I had my suspicions
about this when I noticed the huge difference in time between writing to
/dev/null vs. an actual file during profiling.

On one of our systems, here's the number of syscalls _before_ this change:

  $ strace -c target/release/tameld --emit xmle -o foo foo.xmlo
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   85.05    4.966192          16    318473           write
    7.23    0.421977          13     32298           lstat
    6.53    0.381424          15     25113           read
    0.75    0.043691          13      3350           readlink
    0.25    0.014713          61       241           close
    0.12    0.007167          30       241           openat
    0.05    0.003175         151        21           munmap
    0.01    0.000488          14        35           brk
    0.01    0.000292           9        33           mmap
    0.00    0.000266          38         7           mremap
    0.00    0.000004           1         3           sigaltstack
    0.00    0.000000           0         6           fstat
    0.00    0.000000           0         1           poll
    0.00    0.000000           0        11           mprotect
    0.00    0.000000           0         7           rt_sigaction
    0.00    0.000000           0         1           rt_sigprocmask
    0.00    0.000000           0         6         6 access
    0.00    0.000000           0         1           execve
    0.00    0.000000           0         1           arch_prctl
    0.00    0.000000           0         1           sched_getaffinity
    0.00    0.000000           0         1           set_tid_address
    0.00    0.000000           0         1           set_robust_list
    0.00    0.000000           0         2           prlimit64
  ------ ----------- ----------- --------- --------- ----------------
  100.00    5.839389                379854         6 total

And _after_:

  $ strace -c target/release/tameld --emit xmle -o foo foo.xmlo
  % time     seconds  usecs/call     calls    errors syscall
  ------ ----------- ----------- --------- --------- ----------------
   45.21    0.435010          13     32298           lstat
   40.09    0.385752          15     25113           read
    6.14    0.059113          21      2809           write
    4.75    0.045687          14      3350           readlink
    2.51    0.024115         100       241           close
    0.84    0.008045          33       241           openat
    0.26    0.002468         118        21           munmap
    0.06    0.000580          17        35           brk
    0.06    0.000566          17        33           mmap
    0.03    0.000279          40         7           mremap
    0.02    0.000181          16        11           mprotect
    0.01    0.000087          15         6         6 access
    0.01    0.000082          12         7           rt_sigaction
    0.01    0.000075          13         6           fstat
    0.00    0.000027           9         3           sigaltstack
    0.00    0.000024          12         2           prlimit64
    0.00    0.000018          18         1           execve
    0.00    0.000016          16         1           poll
    0.00    0.000013          13         1           sched_getaffinity
    0.00    0.000012          12         1           rt_sigprocmask
    0.00    0.000012          12         1           arch_prctl
    0.00    0.000012          12         1           set_robust_list
    0.00    0.000011          11         1           set_tid_address
  ------ ----------- ----------- --------- --------- ----------------
  100.00    0.962185                 64190         6 total

What a difference!

There's still a lot of other red flags in there; those can be addressed
separately.

This was originally written as I was learning Rust, and I suspect that I
didn't realize that File wasn't buffered at the time.

For the above link: times go from 1.23s pre-change to 0.85s after:

  0.77user 0.44system 0:01.23elapsed 99%CPU (0avgtext+0avgdata 48520maxresident)k
  0inputs+43952outputs (0major+12825minor)pagefaults 0swaps

  0.69user 0.15system 0:00.85elapsed 98%CPU (0avgtext+0avgdata 48396maxresident)k
  0inputs+43952outputs (0major+12823minor)pagefaults 0swaps
2021-08-20 12:14:42 -04:00
Mike Gerwitz c9a2ae533f tamer: xir (XmlWriter)[write_new]: Correct #[must_use] declaration
The return value has no meaningful side-effects at all; the write operation
failing isn't worth pointing out, since it has to be used regardless.

The normal `write` does have useful side-effects, of course.
2021-08-20 11:38:58 -04:00
Mike Gerwitz 59d578e669 tamer: xir (XmlWriter)[write_new]: New method
This change was primarily intended to clean up unit tests.  Since it
allocates and returns a new buffer, I do not expect this to have much use
within TAMER itself in the near future.  Maybe in later tooling.

If this is abused, person from the future: add `#[cfg(test)]` to its
definition.
2021-08-20 11:37:01 -04:00
Mike Gerwitz cd1eae95ca tamer: xir: {NodeStream=>Token}
I decided not to do this in a previous commit because I had documented
"NodeStream" elsewhere, so I'd like it to be in the Git history to
understand its evolution.

This never was a "Node" stream beyond the initial concept phase, because it
represents tokens that aren't themselves nodes.  It is intended to generate
XML nodes, but may need to accommodate non-nodes (e.g. XML declarations) in
the future.

The name originated from `Node`, which was a tree-based IR that was
initially conceived, but removed because it's not yet needed.  What we need
is a streaming IR for xmle writing, and then for reading and echoing back
out XML for the new frontend.
2021-08-20 10:30:27 -04:00
Mike Gerwitz a23bae5e4d tamer: XIR: Working concept
This is a working streaming IR for XML.  I want to get this committed before
I go further cleaning it up and integrating it into the xmle writer.

This is lacking detailed documentation, and the names of things may end up
changing.

Initial benchmarks do show that it has a ~2x performance improvement over
quick-xml when dealing with two attributes on a node, and I suspect that
improvement will increase with the number of attributes.  We will see how it
compares in real-world benchmarks once the linker has been modified to use
it.

The goal isn't to _avoid_ quick-xml---it'll be used in the future for things
like escaping that would be a huge waste to implement ourselves.  It just so
happened that quick-xml was not beneficial for these changes; indeed, its
own writer is fairly simple for the portions that were implemented here, so
there's no use in fighting with its API, particularly around attributes and
our need to explicitly control whitespace (with the intent of handling code
formatters in the future).

To put this into perspective: the reason this work is being done isn't to
refactor the linker, or to speed it up, but to generalize XML writing and
provide a suitable IR for use in the compiler.  The first step of the
frontend is to essentially echo the XML token stream back out so we can
incrementally parse it and do something useful, to incrementally rewrite the
compiler in Rust.
2021-08-20 10:16:36 -04:00
Mike Gerwitz c211ada89b tamer: benches (memchr): Add missing bench attr
This benchmark was not being run.
2021-08-19 23:14:33 -04:00
Mike Gerwitz e217478a46 tamer: Makefile.am (CARGO_BENCH_FLAGS): New env var 2021-08-19 16:43:14 -04:00
Mike Gerwitz fc235b7ecc tamer: memchr benches
This adds benchmarking for the memchr crate.  It is used primarily by
quick-xml at the moment, but the question is whether to rely on it for
certain operations for XIR.

The benchmarking on an Intel Xeon system shows that memchr and Rust's
contains() perform very similarly on small inputs, matching against a single
character, and so Rust's built-in should be preferred in that case so that
we're using APIs that are familiar to most people.

When larger inputs are compared against, there's a greater benefit (a little
under ~2x).

When comparing against two characters, they are again very close.  But look
at when we compare two characters against _multiple_ inputs:

  running 24 tests
  test large_str:1️⃣:memchr_early_match                 ... bench:       4,938 ns/iter (+/- 124)
  test large_str:1️⃣:memchr_late_match                  ... bench:      81,807 ns/iter (+/- 1,153)
  test large_str:1️⃣:memchr_non_match                   ... bench:      82,074 ns/iter (+/- 1,062)
  test large_str:1️⃣:rust_contains_one_byte_early_match ... bench:       9,425 ns/iter (+/- 167)
  test large_str:1️⃣:rust_contains_one_byte_late_match  ... bench:     123,685 ns/iter (+/- 3,728)
  test large_str:1️⃣:rust_contains_one_byte_non_match   ... bench:     123,117 ns/iter (+/- 2,200)
  test large_str:1️⃣:rust_contains_one_char_early_match ... bench:       9,561 ns/iter (+/- 507)
  test large_str:1️⃣:rust_contains_one_char_late_match  ... bench:     123,929 ns/iter (+/- 2,377)
  test large_str:1️⃣:rust_contains_one_char_non_match   ... bench:     122,989 ns/iter (+/- 2,788)
  test large_str:2️⃣:memchr2_early_match                ... bench:       5,704 ns/iter (+/- 91)
  test large_str:2️⃣:memchr2_late_match                 ... bench:      89,194 ns/iter (+/- 8,546)
  test large_str:2️⃣:memchr2_non_match                  ... bench:      85,649 ns/iter (+/- 3,879)
  test large_str:2️⃣:rust_contains_two_char_early_match ... bench:      66,785 ns/iter (+/- 3,385)
  test large_str:2️⃣:rust_contains_two_char_late_match  ... bench:   2,148,064 ns/iter (+/- 21,812)
  test large_str:2️⃣:rust_contains_two_char_non_match   ... bench:   2,322,082 ns/iter (+/- 22,947)
  test small_str:1️⃣:memchr_mid_match                   ... bench:       4,737 ns/iter (+/- 842)
  test small_str:1️⃣:memchr_non_match                   ... bench:       5,160 ns/iter (+/- 62)
  test small_str:1️⃣:rust_contains_one_byte_non_match   ... bench:       3,930 ns/iter (+/- 35)
  test small_str:1️⃣:rust_contains_one_char_mid_match   ... bench:       3,677 ns/iter (+/- 618)
  test small_str:1️⃣:rust_contains_one_char_non_match   ... bench:       5,415 ns/iter (+/- 221)
  test small_str:2️⃣:memchr2_mid_match                  ... bench:       5,488 ns/iter (+/- 888)
  test small_str:2️⃣:memchr2_non_match                  ... bench:       6,788 ns/iter (+/- 134)
  test small_str:2️⃣:rust_contains_two_char_mid_match   ... bench:       6,203 ns/iter (+/- 170)
  test small_str:2️⃣:rust_contains_two_char_non_match   ... bench:       7,853 ns/iter (+/- 713)

Yikes.

With that said, we won't be comparing against such large inputs
short-term.  The larger strings (fragments) are copied verbatim, and not
compared against---but they _were_ prior to the previous commit that stopped
unencoding and re-encoding.

So: Rust built-ins for inputs that are expected to be small.
2021-08-18 14:23:03 -04:00
Mike Gerwitz 1cdb3fbbc5 tamer: tameld: Skip fragment unescaping only to re-escape on write
Fragments' text were unescaped on reading, producing an owned String and
spending time parsing the text to unescape.  We were then copying that into
an internement pool (so, copying twice, effectively).

Further, we were then _re-escaping_ on write.

This was all wasteful, since we do not do any manipulation of the fragment
before outputting to the xmle file; we know that Saxon produced properly
escaped XML to begin with, and can trust to propagate it.

This also introduces a new global `clone_uninterned_utf8_unchecked` method.

In profiling this change, I tested (a) before this change, (b) after writing
without escaping, and (c) after both reading escaped and writing without
escaping.

     (a)              (b)              (c)
  sec   mem (B)    sec     B        sec     B
0:00.95 47896 -> 0:00.91 47988 -> 0:00.87 48288
0:00.40 30176 -> 0:00.37 25656 -> 0:00.36 25788
0:00.39 45672 -> 0:00.37 45756 -> 0:00.35 34952
0:00.39 20716 -> 0:00.38 19604 -> 0:00.36 19956
0:00.33 16836 -> 0:00.32 16988 -> 0:00.31 16892
0:00.23 15268 -> 0:00.23 15236 -> 0:00.22 15312
0:00.44 20780 -> 0:00.44 20048 -> 0:00.41 20148
0:00.54 44516 -> 0:00.50 36964 -> 0:00.49 36728
0:00.62 55976 -> 0:00.57 46204 -> 0:00.54 41468
0:00.31 28016 -> 0:00.30 27308 -> 0:00.28 23844
0:00.23 15388 -> 0:00.22 15316 -> 0:00.21 15304
0:00.05 4888  -> 0:00.05 4760  -> 0:00.05 4948
0:00.41 19756 -> 0:00.41 19852 -> 0:00.40 19992
0:00.47 20828 -> 0:00.46 20844 -> 0:00.44 20968
0:00.27 18152 -> 0:00.26 18184 -> 0:00.25 18312

Interestingly, the peak memory usage increases very slightly between the
second and third steps (though decreases from the first), likely because the
raw (encoded) is larger than the unencoded text (e.g. `&gt;` takes more
space than `>`).
2021-08-18 11:39:06 -04:00
Mike Gerwitz f97141f5c5 tamer: tameld: Use uninterned symbols for reader
Fragments were previously represented by `String` to avoid the cost of
interning (hashing and copying).  This change modifies it to use uninterned
symbols, which does still have a copy overhead but it does not hash.

Initial tests shows a small performance decrease of about 15% and a small
memory increase of similar proportion.  However, once I realized that I was
not clearing buffers from quick_xml events and implemented that change in a
previous commit, this change ended up being approximately on par with
`String`, despite the copying of some pretty large fragments.

YMMV, though, and perhaps on less powerful systems time may increase
slightly.

The upcoming XIR (XML IR) was originally going to support both owned strings
and symbols, but now we'll just use uninterned symbols; I can't rationalize
complicating the API at this time when it will provide an almost
imperceivable performance benefit.  If ever that changes in the future,
that change will be entertained.

The end result is that the fate of a fragment's underlying memory is
determined by whatever is processing the data, _not_ by the API itself---the
API was previously forcing use of a String, whereas now it's up to the
caller to determine whether we want comparable interns.  For fragments,
that's not likely ever to be the case, especially considering that the
representation will change so drastically in the future.
2021-08-16 14:05:32 -04:00
Mike Gerwitz d96dcad7d8 tamer: tameld: Reduce peak memory usage
This clears the buffers used by quick_xml, which was apparently forgotten
during initial development (I think I expected it to re-use the previously
allocated space automatically).

This has significant effects in some cases.  For example, one of our UI
builds drops from ~9KiB to ~5KiB peak memory usage.  Other builds for larger
suppliers are only slightly effected because of some of their massive
fragments.
2021-08-16 13:38:14 -04:00
Mike Gerwitz ce233ac01d tamer: sym: Uninterned symbols
This adds support for uninterned symbols.  This came about as I was creating
Xir (not yet committed) where I had to decide if I wanted `SymbolId` for all
values, even though some values (e.g. large text blocks like compiled code
fragments for xmle files) will never be compared, and so would be wastefull
hashed.

Previous IRs used `String`, but that was clumsy; see documentation in this
commit for rationale.
2021-08-13 22:54:04 -04:00
Mike Gerwitz 0ff0f88e5f tamer: Introduce span
This is an initial implementation optimized for expected use
cases.  Hopefully that pans out and doesn't come back to bite me.

Regarding the context: it only allows for interned paths atm, which are
strings (and so much be valid UTF-8, which is fine for us, but sucks for
something more general-purpose).  I'll be curious if the context needs
extension later on, or if different contexts will be stored in IRs (e.g. to
store a template application site as well as the location of the expansion
within the template body).
2021-08-13 15:16:39 -04:00
Mike Gerwitz 29ab4b9bfc tamer: sym: Disallow SymbolId construction outside of module
SymboldIds must only be constructed by interners, otherwise we lose
confidence in the type.

This offers an associated function to construct raw SymbolIds from integers
for testing purposes.
2021-08-13 11:54:11 -04:00
Mike Gerwitz d11b4220b2 Revert "tamer: Cargo.toml (dependencies)[lazy_static]: Remove (now used)"
This reverts commit 4fd6313cd2.

...and now I need it for tests.
2021-08-12 16:08:34 -04:00
Mike Gerwitz 4fd6313cd2 tamer: Cargo.toml (dependencies)[lazy_static]: Remove (now used)
The previous commit removed all uses.
2021-08-11 16:26:36 -04:00
Mike Gerwitz 9deb393bfd tamer: Global interners
This is a major change, and I apologize for it all being in one commit.  I
had wanted to break it up, but doing so would have required a significant
amount of temporary work that was not worth doing while I'm the only one
working on this project at the moment.

This accomplishes a number of important things, now that I'm preparing to
write the first compiler frontend for TAMER:

  1. `Symbol` has been removed; `SymbolId` is used in its place.
  2. Consequently, symbols use 16 or 32 bits, rather than a 64-bit pointer.
  3. Using symbols no longer requires dereferencing.
  4. **Lifetimes no longer pollute the entire system! (`'i`)**
  5. Two global interners are offered to produce `SymbolStr` with `'static`
     lifetimes, simplfiying lifetime management and borrowing where strings
     are still needed.
  6. A nice API is provided for interning and lookups (e.g. "foo".intern())
     which makes this look like a core feature of Rust.

Unfortunately, making this change required modifications to...virtually
everything.  And that serves to emphasize why this change was needed:
_everything_ used symbols, and so there's no use in not providing globals.

I implemented this in a way that still provides for loose coupling through
Rust's trait system.  Indeed, Rustc offers a global interner, and I decided
not to go that route initially because it wasn't clear to me that such a
thing was desirable.  It didn't become apparent to me, in fact, until the
recent commit where I introduced `SymbolIndexSize` and saw how many things
had to be touched; the linker evolved so rapidly as I was trying to learn
Rust that I lost track of how bad it got.

Further, this shows how the design of the internment system was a bit
naive---I assumed certain requirements that never panned out.  In
particular, everything using symbols stored `&'i Symbol<'i>`---that is, a
reference (usize) to an object containing an index (32-bit) and a string
slice (128-bit).  So it was a reference to a pretty large value, which was
allocated in the arena alongside the interned string itself.

But, that was assuming that something would need both the symbol index _and_
a readily available string.  That's not the case.  In fact, it's pretty
clear that interning happens at the beginning of execution, that `SymbolId`
is all that's needed during processing (unless an error occurs; more on that
below); and it's not until _the very end_ that we need to retrieve interned
strings from the pool to write either to a file or to display to the
user.  It was horribly wasteful!

So `SymbolId` solves the lifetime issue in itself for most systems, but it
still requires that an interner be available for anything that needs to
create or resolve symbols, which, as it turns out, is still a lot of
things.  Therefore, I decided to implement them as thread-local static
variables, which is very similar to what Rustc does itself (Rustc's are
scoped).  TAMER does not use threads, so the resulting `'static` lifetime
should be just fine for now.  Eventually I'd like to implement `!Send` and
`!Sync`, though, to prevent references from escaping the thread (as noted in
the patch); I can't do that yet, since the feature has not yet been
stabalized.

In the end, this leaves us with a system that's much easier to use and
maintain; hopefully easier for newcomers to get into without having to deal
with so many complex lifetimes; and a nice API that makes it a pleasure to
work with symbols.

Admittedly, the `SymbolIndexSize` adds some complexity, and we'll see if I
end up regretting that down the line, but it exists for an important reason:
the `Span` and other structures that'll be introduced need to pack a lot of
data into 64 bits so they can be freely copied around to keep lifetimes
simple without wreaking havoc in other ways, but a 32-bit symbol size needed
by the linker is too large for that.  (Actually, the linker doesn't yet need
32 bits for our systems, but it's going to in the somewhat near future
unless we optimize away a bunch of symbols...but I'd really rather not have
the linker hit a limit that requires a lot of code changes to resolve).

Rustc uses interned spans when they exceed 8 bytes, but I'd prefer to avoid
that for now.  Most systems can just use on of the `PkgSymbolId` or
`ProgSymbolId` type aliases and not have to worry about it.  Systems that
are actually shared between the compiler and the linker do, though, but it's
not like we don't already have a bunch of trait bounds.

Of course, as we implement link-time optimizations (LTO) in the future, it's
possible most things will need the size and I'll grow frustrated with that
and possibly revisit this.  We shall see.

Anyway, this was exhausting...and...onward to the first frontend!
2021-08-11 14:24:55 -04:00
Mike Gerwitz 71011f5724 tamer: sym: Split into multiple modules
This helps to organize a bit better as I prepare to introduce singleton
interners.
2021-08-02 23:54:37 -04:00
Mike Gerwitz 01722c9c3b tamer: Symbol{Index=>Id}
The former was a misnomer (it represents an index _entry_).  This name is
also shorter, which is nice, considering how often it'll be used.
2021-07-30 13:32:32 -04:00
Mike Gerwitz 0fc8a1a4df tamer: Remove default SymbolIndex (et al) index type
Oh boy.  What a mess of a change.

This demonstrates some significant issues we have with Symbol.  I had
originally modelled the system a bit after Rustc's, but deviated in certain
regards:

  1. This has a confurable base type to enable better packing without bit
     twiddling and potentially unsafe tricks I'd rather avoid unless
     necessary; and
  2. The lifetime is not static, and there is no global, singleton interner;
     and
  3. I pass around references to a Symbol rather than passing around an
     index into an interner.

For #3---this is done because there's no singleton interner and therefore
resolving a symbol requires a direct reference to an available interner.  It
also wasn't clear to me (and still isn't, in fact) whether more than one
interner may be used for different contexts.

But, that doesn't preclude removing lifetimes and just passing around
indexes; in fact, I plan to do this in the frontend where the parser and
such will have direct interner access and can therefore just look up based
on a symbol index.  We could reserve references for situations where
exposing an interner would be undesirable.

Anyway, more to come...
2021-07-29 14:26:40 -04:00
Mike Gerwitz e6ad2be5b9 tamer: sym: Primitive-based SupportedSymbolIndex
As mentioned in the previous commit, this flips the types such that the base
type if the primitive and the associated type is the `NonZero*` type; this
is much more natural, concise, and allows Rust to infer the proper type in
most every situation.

The next step will be to stop defaulting the index type for SymbolIndex and
related, since we are about to care very much what size it is (compiler
vs. linker).
2021-07-28 15:21:24 -04:00
Mike Gerwitz e562d7fcc8 tamer: sym: Begin SymbolIndex base data generalization
This was previously a NonZeroU32, but it was intended to support NonZeroU16
as well for packages, so that we can fit symbols into smaller spaces.  In
particular, the upcoming Span wants to fit within 8 bytes, and so requires a
smaller SymbolIndex type.

I'm unhappy with this current implementation, and so comments are unfinished
and there are a couple ignores for dead code warnings.  I want to flip the
`SupportedSymbolIndex` trait so that users can specify the primitive rather
than the NonZero* type, which is really awkward-looking and verbose,
especially if you have to do `SymbolIndex::<NonZeroU32>::from_int` or
something.  It also prevents (at least in the cases I've observed) Rust from
inferring the proper type for you based on the argument you provide.

So, the goal will be `SymbolIndex::<u32>::from_int(n)`, for example.
2021-07-28 15:21:15 -04:00
Mike Gerwitz ca6ef3ed36 tamer: frontend: Begin basic XML parsing
The first step in the process is to emit the raw XML events that can then be
immediately output again to echo the results into another file.  This will
then allow us to begin parsing the input incrementally, and begin to morph
the output into a real `xmlo` file.
2021-07-27 00:37:13 -04:00
Mike Gerwitz d9dcfe8777 tamer: Introduce tpwrap module to contain quick_xml::Error adapter
This adapter exists to implement PartialEq so that it can be derived on
Error objects.  This is used primarily (well, exclusively atm) for tests.
2021-07-23 23:23:55 -04:00
Mike Gerwitz fb8422d670 tamer: Initial frontend concept
This introduces the beginnings of frontends for TAMER, gated behind a
`wip-features` flag.

This will be introduced in stages:

  1. Replace the existing copy with a parser-based copy (echo back out the
     tokens), when the flag is on.
  2. Begin to parse portions of the source, augmenting the output xmlo (xmli
     at the moment).  The XSLT-based compiler will be modified to skip
     compilation steps as necessary.

As portions of the compilation are implemented in TAMER, they'll be placed
behind their own feature flags and stabalized, which will incrementally
remove the compilation steps from the XSLT-based system.  The result should
be substantial incremental performance improvements.

Short-term, the priorities are for loading identifiers into an IR
are (though the order may change):

  1. Echo
  2. Imports
  3. Extern declarations.
  4. Simple identifiers (e.g. param, const, template, etc).
  5. Classifications.
  6. Documentation expressions.
  7. Calculation expressions.
  8. Template applications.
  9. Template definitions.
  10. Inline templates.

After each of those are done, the resulting xmlo (xmli) will have fully
reconstructed the source document from the IR produced during parsing.
2021-07-23 22:24:08 -04:00
Mike Gerwitz 60372d2960 tamer: Makefile.am (all): Binaries and doc
`all` was previously the target for binaries only.
2021-07-23 22:23:10 -04:00
Mike Gerwitz 6ec1a49506 tamer: Makefile.am: Include feature flags for doc generation and tests
This was forgotten in the previous commit.
2021-07-23 15:56:33 -04:00
Mike Gerwitz f1a3273ee3 tamer: configure.ac: Configure-time feature flags (via Cargo) 2021-07-23 10:16:44 -04:00
Mike Gerwitz 5aaa1106cb tamer: obj::xmlo::reader::mock: Extract into crate::test::quick_xml
Other mocks exist here, and here it can be re-used for the upcoming XML
frontend.
2021-07-22 15:32:30 -04:00
Mike Gerwitz 2e50af1220 Copyright year update 2021 2021-07-22 15:00:15 -04:00
Mike Gerwitz e5bbd49166 tamer: obj::xmlo::reader: Extract tests separate file
The file's getting a bit large and the tests are rather complex.  Further,
LSP does better on smaller, less complex files.
2021-07-22 14:39:06 -04:00
Mike Gerwitz 1f24cfdf25 Remove :map: sym-dep generation
This was incorrect to begin with---it does not make sense that an input
mapping should depend upon the identifier that it maps to, in the sense that
we make use of these dependencies.  If we add weak symbol references in the
future, then this can be reintroduced.

By removing this, we free tameld from having to perform the check itself.

.rev-xmlo bumped to force rebuilding of object files since the linker now
expects that no such dependencies will exist within them.
2021-07-22 14:27:15 -04:00
Mike Gerwitz 90c6b51fd5 tamer: tameld: Place constants into static section in executable
This is something that changed when the TAMER POC was initially created, as
I was learning Rust.  I don't recall the original reason why this was moved,
but it could have been moved back long ago.

In our systems, constants can hold tables (as matrices) with tens or
hundreds of thousands of rows, and there are a number of them in certain
projects.  As an example, the YAML-based test cases for one of our systems
went from ~2m30s to ~45s after this change was made.  Much of the cost
savings comes from saving GC.
2021-07-21 14:53:15 -04:00
Mike Gerwitz 93fb1f1bdd tamer: Rust v1.{48=>53}.0 for rustdoc tool lints
A previous commit used a rustdoc tool lint, but that support wasn't added
until 1.52.0 (2021-05-06).

Note that this represents the minimum _required_ version to build TAMER; you
can use a later version.
2021-06-22 09:07:53 -04:00
Mike Gerwitz 716556c39f tamer: Rust 1.{42=>48}.0 for stable intra-doc links without nightly 2021-06-21 13:10:00 -04:00
Mike Gerwitz 96ea0302cc tamer: Cargo.lock: Dependency updates
This project has been on pause for over a year.
2021-06-21 12:46:38 -04:00
Mike Gerwitz 96ffd5f6e5 [DEV-8000] ir::asg: Error types for unresolved identifiers during sorting
This checks explicitly for unresolved objects while sorting and provides an
explicit error for them.  For example, this will catch externs that have no
concrete resolution.

This previously fell all the way through to the unreachable! block.  The old
POC implementation was catching unresolved objects, albeit with a debug
error.
2020-07-02 01:38:32 -04:00
Mike Gerwitz a2415c8c6f [DEV-8000] ir::asg::base: Replace Symbol::new_dummy
Use symbol_dummy!.
2020-07-01 15:53:56 -04:00
Mike Gerwitz 0d4bbe5e4e [DEV-8000] ir::asg: Introduce SortableAsgError
This will be used for the next commit, but this change has been isolated
both because it distracts from the implementation change in the next commit,
and because it cleans up the code by removing the need for a type parameter
on `AsgError`.

Note that the sort test cases now use `unwrap` instead of having
`{,Sortable}AsgError` support one or the other---this is because that does
not currently happen in practice, and there is not supposed to be a
hierarchy; they are siblings (though perhaps their name may imply otherwise).
2020-07-01 13:42:14 -04:00
Mike Gerwitz f832feb3fa [DEV-8000] ir::asg::base::BaseAsg::check_cycles: Extract into function
The only reason this function was a method of `BaseAsg` was because of
`self.graph`, which is accessible within the scope of this
module.  `check_cycles` is logically associated with `SortableAsg`, and so
should exist alongside it (though it can't exist as an associated function
of that trait).
2020-07-01 11:02:20 -04:00
Joseph Frazer 43d00a8268 [DEV-7504] Add GraphML generation
We want to be able to build a representation of the dependency graph so
we can easily inspect it.

We do not want to make GraphML by default. It is better to use a tool.
We use "petgraph-graphml".
2020-05-13 08:04:48 -04:00
Mike Gerwitz 0127d4b698 TAMER: sym::Interner::index_lookup
This was originally omitted because there wasn't a use case for it.  Now
that we're adding context to errors, however, an owned value is highly
desirable.

This adds almost no measurable overhead to the internment system in
benchmarks (largely within the margin of error).
2020-04-29 11:33:41 -04:00
Mike Gerwitz 4b643385c8 TAMER: Update Cargo dependencies 2020-04-29 11:33:38 -04:00
Mike Gerwitz bcca5f7c49 [DEV-7084] TAMER: AsgBuilder and IR lowering docs 2020-04-28 13:39:55 -04:00
Mike Gerwitz 0f4b2d75f8 [DEV-7084] TAMER: obj::xmlo: Private inner modules 2020-04-28 11:08:05 -04:00
Mike Gerwitz 549e9ca23b [DEV-7084] TAMER: AsgBuilderState:🆕 New constructor 2020-04-28 09:06:25 -04:00
Mike Gerwitz 9893d56775 [DEV-7084] TAMER: Finalize AsgBuilder 2020-04-28 09:06:25 -04:00
Mike Gerwitz 32abc7dce2 [DEV-7084] TAMER: impl PartialEq for XmloError
This cannot be dervied because XmlError does not implement PartialEq,
which is quite the annoyance in tests.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 21a0bdcce1 [DEV-7084] TAMER: AsgBuilderError: Introduce proper error variants
This is a union (sum type) of three other errors types, plus errors specific
to this builder.

This commit does a good job demonstrating the boilerplate, as well as a need
for additional context (in the case of `IdentKindError`), that we'll want to
work on abstracting away.
2020-04-28 09:06:25 -04:00
Mike Gerwitz ef79a763ac [DEV-7084] TAMER: Correct Ix trait bound for AsgError
The `Debug` bound is inconvenient and requires propagation to any types that
use it.  Further, it's really awkward having `Display` depend on `Debug`; if
we want to render a useful display here, we can write one.

To be clear: IndexType implements Debug.

For now, this is pretty-printed by another part of the code, which we don't
want to implement in `Display` because it requires looking things up from
the graph.
2020-04-28 09:06:25 -04:00
Mike Gerwitz cfc13f9016 [DEV-7084] TAMER: ir::asg::IdentKindError: Replace string with enum 2020-04-28 09:06:25 -04:00
Mike Gerwitz 0a9a3214b7 [DEV-7084] TAMER: ir::asg::BaseAsg:🆕 New associated function
Profiling showed that creating an initial capacity of 0 did not have a
notable affect on performance.
2020-04-28 09:06:25 -04:00
Mike Gerwitz ecc2e33ba7 [DEV-7084] TAMER: xmlo::AsgBuilder: Accept XmloResult iterator
This flips the API from using XmloWriter as the context to using Asg and
consuming anything that can produce XmloResults.  This not only makes more
sense, but avoids having to create a trait for XmloReader, and simplifies
the trait bounds we have to concern ourselves with.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 323ea79bf8 [DEV-7084] TAMER: Basic AsgBuilder cleanup
This just tidies things up a little bit before I get into some further
refactoring.  I wrote the original code when I was just learning Rust not
too long ago, so it's interesting to see how my understanding has changed
over that relatively short period of time.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 9220de4769 [DEV-7084] TAMER: Finish encapsulating petgraph
This will allow us to migrate away from Petgraph in the future should we
choose to do so.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 0f423f3b24 [DEV-7084] TAMER: Simplify path canonicalization
This abstracts away the canonicalizer and solves the problem whereby
canonicalization was not being performed prior to recording whether a path
has been visited.  This ensures that multiple relative paths to the same
file will be properly recognized as visited.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 4a7e00c404 [DEV-7084] TAMER: ld::poc: Remove unused fragments arg 2020-04-28 09:06:25 -04:00
Mike Gerwitz c94120335f [DEV-7084] TAMER: ld::poc: Remove unnecessary initial path canonicalization
Less to refactor and test.
2020-04-28 09:06:25 -04:00
Mike Gerwitz da69118592 [DEV-7084] TAMER: AsgBuilderState
This completes the POC extraction for AsgBuilder, but is still POC
code.  The commits that follow will clean it up and provide tests.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 3f46917da9 [DEV-7084] TAMER: AsgBuilder extracted from POC
This extracts the changes nearly verbatim before doing refactoring so that
it's easier to observe what changes have been made.
2020-04-28 09:06:25 -04:00
Mike Gerwitz 7ed0691c45 [DEV-7084] TAMER: fs: impl File for BufReader
This further simplifies the POC linker.
2020-04-28 09:06:25 -04:00
Mike Gerwitz fbfb3c4ba2 [DEV-7084] TAMER: CanonicalFile
This will be entirely replaced in an upcoming commit.  See that for
details.  I don't feel like dealing with the conflicts for rearranging and
squashing these commits.
2020-04-28 09:06:25 -04:00
Mike Gerwitz d97e53a835 [DEV-7084] TAMER: fs: Basic filesystem abstraction
This also includes an implementation to visit paths only once.  Note that it
does not yet canonicalize the path before visiting, so relative paths to the
same file can slip through, and relative paths to _different_ files could be
erroneously considered to have been visited.

This will be fixed in an upcoming commit.
2020-04-28 09:06:19 -04:00
Mike Gerwitz 90ed4e9bd6 [DEV-7084] TAMER: From<B, &I> for XmloReader
This serves as a constructor for the time being, decoupling from POC.  We
may do something better once we have a better idea of how the various
abstractions around this will evolve.
2020-04-20 10:53:51 -04:00
Joseph Frazer 2c587e2d9d [DEV-7147] Add "tamec" executable
Add a stub executable that will eventually become a full-featured TAME
compiler. The first implementation will only copy the source file to an
intermediary file that will be compiled by the XSLT compiler.
2020-04-09 09:46:46 -04:00
Mike Gerwitz 8385b64e1d [DEV-7086] TAMER: Remove WIP linker warning
While it is true that this is still being finalized, the warnings originally
existed because tameld was not feature complete.  It is now.
2020-04-06 10:04:19 -04:00
Mike Gerwitz 68c7636be8 [DEV-7086] TAMER: ir::asg::base::test Add missing set_fragment failure test
Results the last remaining BaseAsg test TODO.
2020-04-06 09:56:13 -04:00
Mike Gerwitz b870480944 [DEV-7086] TAMER: ir::asg::TransitionError::BadFragmentDest tuple=>struct
Consistency.
2020-04-06 09:56:13 -04:00
Mike Gerwitz da5057058d [DEV-7086] TAMER: Disallow IdentObject::resolve redeclarations
Except under well-defined circumstances.
2020-04-06 09:56:12 -04:00
Mike Gerwitz 0868453dab [DEV-7086] Proper handling of identifier overrides
This is an awkward system that I'd like to remove at some point.  It adds
complexity.  For the meantime, overrides have been arbitrarily restricted to
a single override (no override-override).  But it's needed being until we
rework maps and can handle the illusion of overrides using the template
system.
2020-04-06 09:55:54 -04:00
Mike Gerwitz a4657580ca [DEV-7086] TAMER: TransitionError::Incompatible: Remove unused 2020-04-01 15:56:33 -04:00
Mike Gerwitz 0f9acd16cd [DEV-7086] TAMER: BaseAsg::set_fragment: Remove duplicate code
Benchmark performance for this method is still substantially slower.  And
oddly, this nearly doubled the speed of the other two calls (granted, at
that speed, it doesn't matter).
2020-03-31 14:56:34 -04:00
Mike Gerwitz f7ed0dbff3 [DEV-7086] ASG benchmarks 2020-03-31 14:18:26 -04:00
Mike Gerwitz 7c65d729aa TAMER: BaseAsg test: Remove fulfilled stub TODO 2020-03-26 16:16:51 -04:00
Mike Gerwitz 4051debad2 [DEV-7087] TAMER: Add Source to IdentObject::Extern
All of these refactoring commits to arrive at this one final change: the
ability to store the source location for externs so that we can report on
what package is expecting an identifier to be defined.

Phew.  Goodnight.
2020-03-26 09:22:21 -04:00
Mike Gerwitz f44549d730 [DEV-7087] TAMER: Object{State,Data}: API representative of state transitions
The API now enforces beginning at Missing and transitioning through
states.  Methods have been renamed to reflect this.
2020-03-26 09:22:17 -04:00
Mike Gerwitz d3ecd7b228 [DEV-7087] TAMER: BaseAsg: Refactor duplicate declare{,_extern} code 2020-03-26 09:21:50 -04:00
Mike Gerwitz 40eaeb3dc8 [DEV-7087] TAMER: Remote optional Source from ASG and Object
This undoes work I did earlier today...but now we'll be able to support a
Source on an extern.

There is duplicate code between `BaseAsg::declare{,_extern}` that will be
resolved in an upcoming commit.  Upcoming commits will also simplify
terminology and clean up methods on ObjectState.
2020-03-26 09:18:08 -04:00
Mike Gerwitz 7dd8717f2f [DEV-7087] TAMER: Asg: Reintroduce declare_extern
There is some duplication here with `declare` that will be cleared up in a
following commit.  Reintroducing this method is necessary so that Source can
be used to represent the source location of the extern itself; it's
currently None to indicate an extern in `declare`.
2020-03-26 09:15:59 -04:00
Mike Gerwitz 537d9e64af [DEV-7087] TAMER: ObjectState: Introduce extern transition
This is the first step in a more incremental refactoring that previous
commits to undo the optional Source in `ObjectState::ident`.  This provides
an explicit transition to an extern, with the intent of requiring an initial
missing state.  This will simplify logic on the ASG.

Note that the Source provided to this new method is not yet used.  That too
will come in a following commit and will represent the source of the defined
extern rather than the concrete identifier.
2020-03-26 09:14:29 -04:00
Mike Gerwitz d6762ab547 [DEV-7087] TAMER: Type compatability check during extern resolution
This properly verifies extern types, and cleans up Asg's API a little so
that externs aren't handled much differently than other declarations.

With that said, after making src optional, I realized that we will indeed
want source information for externs themselves so we can direct the user to
what package is expecting that symbol (as the old linker does).  So this
approach will not work, and I'll have to undo some of those changes.
2020-03-26 09:14:26 -04:00
Mike Gerwitz 7a972465ea [DEV-7087] TAMER: tameld: Format error output
We will want an option for verbose debug output in the future.
2020-03-26 09:08:13 -04:00
Mike Gerwitz 05d03dc4bb [DEV-7087] Beginning of extern type verification and reporting
This only verifies when externs are defined _before_ they need to be
resolved.  See a future commit for the rest of this.
2020-03-26 09:08:13 -04:00
Mike Gerwitz b35dd4f4dd [DEV-7087] TAMER: AsgError: Wrap TransitionError
See next commit.
2020-03-26 09:08:10 -04:00
Joseph Frazer 6386e096b4 [DEV-7133] Clearly show the cycles in the output 2020-03-26 08:48:43 -04:00
Joseph Frazer 8af93d9339 [DEV-7133] Check for cyclic dependencies
We want the linker to show an error when a cyclic dependency is
encountered.

Co-authored-by: Mike Gerwitz <mike.gerwitz@ryansg.com>
2020-03-26 08:48:43 -04:00
Joseph Frazer 59f194a46a [DEV-7133] Add AsgError::Cycle
We want a special error type when we detect cyclic dependencies.
2020-03-26 08:48:43 -04:00
Mike Gerwitz 7a4f6cf9f2 [DEV-7087] TAMER: symbol_dummy! macro 2020-03-24 14:14:05 -04:00
Mike Gerwitz f969877324 [DEV-7087] TAMER: {=>Ident}Object{,State,Data}
This is essential to clarify what exactly the different object types
represent with the new generic abstractions.  For example, we will have
expressions as an object type.
2020-03-24 09:56:25 -04:00
Mike Gerwitz 5fb68f9b67 TAMER: Make Asg generic over object
There's a lot here to make the object stored on the `Asg` generic.  This
introduces `ObjectState` for state transitions and `ObjectData` for pure
data retrieval.  This will allow not only for mocking, but will be useful to
enforce compile-time restrictions on the type of objects expected by the
linker vs. the compiler (e.g. the linker will not have expressions).

This commit intentionally leaves the corresponding tests in their original
location to prove that the functionality has not changed; they'll be moved
in a future commit.

This also leaves the names as "Object" to reduce the number the cognative
overhead of this commit.  It will be renamed to something like "IdentObject"
in the near future to clarify the intent of the current object type and to
open the way for expressions and a type that marries both of them in the
future.

Once all of this is done, we'll finally be able to make changes to the
compatibility logic in state transitions to implement extern compatibility
checks during resolution.

DEV-7087
2020-03-24 09:56:20 -04:00
Mike Gerwitz f20120787f TAMER: Extract identifier transitions into Object
The next commit will generalize this further.  This moves logic out of
BaseAsg so that we can implement more sophisticated transitions for
compatability checks.

The logic is still tested as part of BaseAsg; the next commit will change
that as it's generalized further.

* tamer/src/ir/asg/base.rs: Extract object transitions.
* tamer/src/ir/asg/graph.rs (AsgError)[IncompatibleIdent]: New variant.
  (From<TransitionError> for AsgError): Basic type translation.
* tamer/src/ir/asg/object.rs (TransitionResult): New type.
  (impl Object): Transition methods.
  (TransitionError): New enum.
2020-03-19 15:42:06 -04:00
Mike Gerwitz 3fe3fc4b84 TAMER: ld/poc: Simplify {get_interner_value=>get_ident} 2020-03-19 15:42:06 -04:00
Mike Gerwitz 400d5b25a1 ir::asg::Object::Empty: Remove variant
This variant is unnecessary, as it was used only by the indexer to represent
the absence of a node, for which was can simply use `None` in the containing
`Option`.

* tamer/Cargo.toml: Add `lazy_static`.
* tamer/Cargo.lock: Update.
* tamer/src/ir/asg/base.rs (with_capacity): Use `None` in place of
    `Some(Object::Empty)`.
* tamer/src/ir/asg/object.rs: Adjust state machine graphic.
  (Empty): Remove variant.
  (Missing): Remove reference to variance.
* tamer/src/lib.rs: Import `lazy_static` for test builds.
* tamer/obj/xmle/writer/writer.rs (Section::iter): Remove `Object::Empty`
    from documentation.
  (test::): Remove references to `Object::Missing`.  `lazy_static!` used
    here.
* tamer/obj/xmle/writer/xmle.rs (test::write_section_catch_missing): Replace
    reference to `Object::Missing`.
2020-03-19 15:42:06 -04:00
Mike Gerwitz 0a135ad707 TAMER: Tidy up graph_sort test
This still isn't comprehensive.  Further, it won't be able to be, because
we'd have to rely on Petgraph implementation details: there are potentially
many acceptable orderings for a given graph.
2020-03-13 11:51:59 -04:00
Joseph Frazer 7e95394076 [DEV-7085] Create `SortableAsg` trait
Create a trait that sorts a graph into `Sections` that can then be used
as an IR. The `BaseAsg` should implement the trait using what was
originally in the POC.
2020-03-13 11:51:59 -04:00
Joseph Frazer bc760387f6 [DEV-7085] Implement `PartialEq` for `Sections`
We want to be able to easily compare `Sections` in tests, so
implementing `PartialEq` (and `Debug`) for both `Sections` and `Section`
is required.
2020-03-13 11:51:59 -04:00
Joseph Frazer 59a0c382af [DEV-7085] Move sections to IR module
We need to use `Sections` in both the writer and the ASG so it needs to
be in a place that makes sense.
2020-03-13 11:51:59 -04:00
Joseph Frazer b5f6a082dd [DEV-7134] Remove unnecessary node replacement
The node was being replaced before we were catching errors properly. Now
that they are propagated, we should not need the replacement.
2020-03-09 11:41:11 -04:00
Joseph Frazer 01e7d3e560 [DEV-7134] Propagate errors from the writer
When an error occurs during the XML writing, they should be shown to the
user.
2020-03-09 08:23:13 -04:00
Joseph Frazer f373a00a80 [DEV-7134] Propagate sorting errors
If a node is found while sorting that is not expected, we should show
the error to the user.
2020-03-09 08:23:13 -04:00
Joseph Frazer 2a5551a04a [DEV-7134] Propagate errors setting fragments
If we cannot set a fragment, we need to display the error to the user.

We are currently ignoring "___head", "___tail", and objects that are
both virtual and overridden. Those will be corrected in with future
changes.
2020-03-09 08:23:13 -04:00
Joseph Frazer 06bc89a9ce [DEV-7134] Pass read event errors up the stack 2020-03-06 14:08:55 -05:00
Joseph Frazer 246a40a047 [DEV-7134] Return error for XmloEvent::SymDecl
We want more than warnings when a XmloEvent::SymDecl symbol has an
unknown "kind".
2020-03-06 13:41:32 -05:00
Joseph Frazer 2228a6158a [DEV-7134] Add alias for LoadResult
It looks better and was recommended by Rust's linter.
2020-03-06 12:44:22 -05:00
Joseph Frazer 4810e7a099 [DEV-7134] Remove unwrap so we can bubble up error messages 2020-03-06 12:32:42 -05:00
Joseph Frazer 590245e191 [DEV-7134] Escalate the error from finding the absolute path
We do not want to have a panic here. The error should be displayed
properly.
2020-03-06 12:24:45 -05:00
Mike Gerwitz bfea768f89 Copyright year 2020 update 2020-03-06 11:05:18 -05:00
Joseph Frazer 4941a7602f [DEV-7081] Add options to tameld
Merge branch 'jira-7081'

* jira-7081:
  [DEV-7081] Add options to tameld
2020-03-06 10:04:48 -05:00
Joseph Frazer e613bd8a8c [DEV-7081] Add options to tameld
We want to add an option to set the output file to the linker so we do
not need to redirect output to awk any longer.

This also adds integration tests for tameld.
2020-03-06 09:41:55 -05:00
Mike Gerwitz 8555cf1e4a configure.ac: Missing cargo-doc error=>warning
Documentation does not need to be built by most users,
who are simply trying to bootstrap the system.
2020-03-05 11:16:15 -05:00
Joseph Frazer 6ac7641087 [DEV-7083] TAMER: xmle writer
This introduces the writer for xmle files.
2020-03-03 11:21:18 -05:00
Mike Gerwitz c2e6efc0b5 TAMER: Additional crate::ld documentation 2020-03-02 15:54:36 -05:00
Mike Gerwitz b89408e5bb TAMER: Extract quick_xml event-related mocks 2020-02-26 10:49:01 -05:00
Mike Gerwitz 19a6d67dc4 TAMER: Separate static xmle section 2020-02-26 10:49:01 -05:00
Mike Gerwitz 7c60b53de8 TAMER: Virtual symbol override 2020-02-26 10:49:01 -05:00
Mike Gerwitz ab3aec980d TAMER: POC: Use FxHash to remove nondeterminism
The default SipHash is a cryptographic hash and causes ordering to change
between runs.
2020-02-26 10:49:00 -05:00
Mike Gerwitz 645908e258 TAMER: xmle output changes to support Summary Page
Co-Authored-By: Joseph Frazer <joseph.frazer@ryansg.com>
2020-02-26 10:49:00 -05:00
Mike Gerwitz 6939753ca0 TAMER: POC: Output xmle
This is a working proof-of-concept that will be finalized in future commits.
2020-02-26 10:49:00 -05:00
Mike Gerwitz 85a4934db5 TAMER: Symbol source data and metadata 2020-02-26 10:49:00 -05:00
Mike Gerwitz bcc2ab1221 TAMER: Initial abstract semantic graph (ASG)
This begins to introduce the ASG, backed by Petgraph.  The API will continue
to evolve, and Petgraph will likely be encapsulated so that our
implementation can vary independently from it (or even remove it in the
future).
2020-02-26 10:48:59 -05:00
Mike Gerwitz f177b6ae5d configure.ac: Rust 1.{39>41}.0 version bump
Relaxes orphan rules for foreign traits.

This also modifies the error to suggest how to update using rustup.
2020-02-25 16:46:28 -05:00
Mike Gerwitz 10b9caa7ad TAMER: Fail on empty fragment ids (and fix underlying problem) 2020-02-25 16:46:28 -05:00
Mike Gerwitz a0893da577 TAMER: xmlo: Add Package event 2020-02-25 16:46:27 -05:00
Mike Gerwitz a8726918f7 TAMER: poc: Use xmlo reader
TODO: More information
2020-02-25 16:46:27 -05:00
Mike Gerwitz a929c8cae4 TAMER: xmlo reader
This introduces the reader for xmlo files produced by the XSLT-based
compiler.  It is an initial implementation but is not complete; see future
commits.
2020-02-25 16:46:25 -05:00
Mike Gerwitz db52fcdb30 Makefile.am (html-am): Add --document-private-items
This generated documenation is only going to be read be developers,
and the private information is very useful to them.
2020-02-25 16:10:57 -05:00
Mike Gerwitz 6aae741162 TAMER (sym::Interner::intern_utf8_unchecked): New function
This removes boilerplate for reading xmlo files.  See next commit.
2020-02-25 16:10:55 -05:00
Mike Gerwitz e8cd378d59 TAMER: Display for Symbol
One of the benefits of storing a reference to the interned string on the
symbol itself is that we get to get its underlying value essentially for
free.
2020-02-24 14:56:28 -05:00
Mike Gerwitz ff0c8bb34f Order symtable, sym-dep, fragments
This ordering will simplify streaming processing of xmlo files in
TAMER.  Specifically, we know that symbols will have been declared by the
time dependencies are added to the graph (and so we should only be creating
edges to existing nodes); and we can halt reading as soon as the closing
fragments tag is encountered, avoiding parsing the entirety of these massive
XML files.

On one particularly large program, this cuts time down from ~0.333s to
~0.300 in the POC linker.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 1f4db84f24 TAMER: Arena-based string interner
Contrary to what I said previously, this replaces the previous
implementation with an arena-backed internment system.  The motivation for
this change was investigating how Rustc performed its string interning, and
why they chose to associate integer identifiers with symbols.

The intent was originally to use Rustc's arena allocator directly, but that
create pulled in far too many dependencies and depended on nightly
Rust.  Bumpalo provides a very similar implementation to Rustc's
DroplessArena, so I went with that instead.

Rustc also relies on a global, singleton interner.  I do not do that
here.  Instead, the returned Symbol carries a lifetime of the underlying
arena, as well as a pointer to the interned string.

Now that this is put to rest, it's time to move on.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 176d099fb6 tamer::sym: FNV => Fx Hash
For strings of any notable length, Fx Hash outperforms FNV.  Rustc also
moved to this hash function and noticed performance
improvements.  Fortunately, as was accounted for in the design, this was a
trivial switch.

Here are some benchmarks to back up that claim:

test hash_set::fnv::with_all_new_1000                 ... bench:     133,096 ns/iter (+/- 1,430)
test hash_set::fnv::with_all_new_1000_with_capacity   ... bench:      82,591 ns/iter (+/- 592)
test hash_set::fnv::with_all_new_rc_str_1000_baseline ... bench:     162,073 ns/iter (+/- 1,277)
test hash_set::fnv::with_one_new_1000                 ... bench:      37,334 ns/iter (+/- 256)
test hash_set::fnv::with_one_new_rc_str_1000_baseline ... bench:      18,263 ns/iter (+/- 261)
test hash_set::fx::with_all_new_1000                  ... bench:      85,217 ns/iter (+/- 1,111)
test hash_set::fx::with_all_new_1000_with_capacity    ... bench:      59,383 ns/iter (+/- 752)
test hash_set::fx::with_all_new_rc_str_1000_baseline  ... bench:      98,802 ns/iter (+/- 1,117)
test hash_set::fx::with_one_new_1000                  ... bench:      42,484 ns/iter (+/- 1,239)
test hash_set::fx::with_one_new_rc_str_1000_baseline  ... bench:      15,000 ns/iter (+/- 233)
test hash_set::with_all_new_1000                      ... bench:     137,645 ns/iter (+/- 1,186)
test hash_set::with_all_new_rc_str_1000_baseline      ... bench:     163,129 ns/iter (+/- 1,725)
test hash_set::with_one_new_1000                      ... bench:      59,051 ns/iter (+/- 1,202)
test hash_set::with_one_new_rc_str_1000_baseline      ... bench:      37,986 ns/iter (+/- 771)
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0d2bb5de59 Makefile.am (clean): New target
Not sure how I missed this one.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 541fbffc2e tameld: Move documentation to tamer::ld 2020-02-24 14:56:28 -05:00
Mike Gerwitz f2b24e6505 HashMapInterner: New interner, docs, and benchmarks
This interner will be suitable for providing an index to look up nodes in
the ASG.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 9a98644213 TAMER: sym::tests: Generate with macro
This will be used for generating the common tests between HashSet and
HashMap implementations.

This is my first macro in Rust.  There does not seem to be a way to
concatenate identifiers (!), so I'm placing them within modules
instead.  That ended up working out just fine, since then I can use a type
to provide the SUT.
2020-02-24 14:56:28 -05:00
Mike Gerwitz e4e0089815 TAMER: Initial string interning abstraction
This is missing two key things that I'll add shortly: a HashMap-based one
for use in the ASG for node mapping, and an entry-based system for
manipulations.

This has been a nice start for exploring various aspects of Rust
development, as well as conventions that I'd like to implement.  In
particular:

  - Robust documentation intended to guide people through learning the
    necessary material about the compiler, as well as related work to
    rationalize design decisions;
  - Benchmarks;
  - TDD;
  - And just getting used to Rust in general.

I've beat this one to death, so I'll commit this and make smaller changes
going forward to show how easily it can evolve.

(This module was originally named `intern` but this commit and those that
follow rewrote it to `sym`.)
2020-02-24 14:56:28 -05:00
Mike Gerwitz 593faa3491 Makefile.am (html-am): Run doc tests
Ensure that we have good examples before generating docs.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 3248c429fe Makefile.am (doc, html): Use intra_rustdoc_links
This is enabled by default in nightly, and is not available at all in
stable.  Considering the PITA that it will be to go back and rewrite docs to
use the new format, and how important of a feature this is, we will just
make use of it now.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0147cb7cb4 Makefile.am (bench): New target
The configure script will determine if nightly is required for running
benchmarks, because `test` is currently an unstable feature.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0acc21f16f Makefile.am (check): Check whether formatting is required
Given that developers should be doing TDD and therefore running this target
frequently, this has the effect of providing immediate feedback when
formatting is needed and outputting a diff.  Developers will then quickly
understand what changes need to be made to avoid future issues (and can run
`cargo fmt` to fix it), at which point they'll rarely ever encounter
formatting errors.

The original purpose was to ensure pipelines fail when the formatter has not
been run.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 3cb67109ec Cargo.toml (profile.release)[lto]: Enable 2020-01-02 10:40:52 -05:00
Mike Gerwitz 8455a38a1d Graph-based POC
This makes use of Petgraph for representing the dependency graph and uses a
separate data structure for both string interning and indexing by symbol
name.
2019-12-02 10:05:48 -05:00
Mike Gerwitz d78d81d721 Cargo.toml: Add petgraph
This will be used to represent the dependency graph.
2019-12-02 10:00:53 -05:00
Mike Gerwitz 717375a84a Cargo.toml: Tame {on=>in} Rust
Changed to match README.md.  This makes more sense too.
2019-12-02 10:00:53 -05:00
Mike Gerwitz 8374541965 tamer: Initial baisc POC with no XML output
This is garbage code.  Do not use it.  It is intentionally throwaway.

While I've researched Rust, I haven't actually _used_ it for a project, so
this is a combination of me exploring various ways of accomplishing the
problem and forcing myself to learn certain aspects of the language.

I'll likely be using petgraph, and this also currently lacks symbol
abstractions.  This commit also performs far too much heap allocation
copying strings around.  But it _does_ perform the topological sort.

Since this only stores the symbol name, it lacks enough information about
the symbol to perform a proper linking.
2019-12-02 10:00:53 -05:00
Mike Gerwitz e53482f2a3 Introduce CARGO_BUILD_FLAGS
This is intended to permit passing `--release`, since dev builds are
terribly slow (e.g. 6s -> 0.2s).  See README.md for more information.
2019-12-02 10:00:49 -05:00
Mike Gerwitz 01e3c33b58 tamer/Cargo.toml: Add quick_xml 2019-11-27 09:16:00 -05:00
Mike Gerwitz e52dd45872 tamer/rustfmt (max_width): Set to 80 2019-11-27 09:15:15 -05:00
Mike Gerwitz c4a8eac59e Makefile.am: Clean up currently-unused path_ vars
Cargo handles it for us.
2019-11-20 10:11:00 -05:00
Mike Gerwitz 7412a8934c tameld: Placeholder binary 2019-11-20 10:11:00 -05:00
Mike Gerwitz f72ff973a7 Makefile.am (all): {cargo=>@CARGO@}
Typo.
2019-11-20 10:11:00 -05:00
Mike Gerwitz f0ca5c60c9 Makefile.am (doc, html): New documentation target 2019-11-20 10:11:00 -05:00
Mike Gerwitz fd1a5837ba TAMER: Initial commit 2019-11-18 14:05:47 -05:00