Commit Graph

1017 Commits (df328da71f296c71240b33c47b283a1825999cf1)

Author SHA1 Message Date
Mike Gerwitz df328da71f tamer: ir::asg::SortableAsg: Move into ld::xmle::lower
This has always been a lowering operation, but it was not phrased in terms
of it, which made the process a bit more confusing to understand.

The implementation hasn't changed, but this is an incremental refactoring
and so exposes BaseAsg and its `graph` field temporarily.

DEV-10859
2021-10-12 09:49:33 -04:00
Mike Gerwitz 81ec65742a tamer: {ir::asg=>ld::xmle}::section
Sections, as written, are specific to xmle files.

I think the intent originally was to have this be more generic, but that
doesn't really make sense.

By explicitly coupling it with `xmle` files, that will allow us to turn this
into a proper lowering operation with its own validations that will allow
`xmle::xir` to do its job without having to validate anything itself.
2021-10-12 00:05:44 -04:00
Mike Gerwitz 1c181b568d tamer: ld::poc: Update comment reflecting current state
The linker is feature-complete, but this file has lived on because the
project was on pause for quite some time.
2021-10-11 23:54:24 -04:00
Mike Gerwitz f899ac898e tamer: {obj=>ld}::xmle
This is a linker-specific module.
2021-10-11 23:52:59 -04:00
Mike Gerwitz 5ea5cffd09 tamer: relroot String->SymbolId
This was [one of] the last remaining Strings; SymbolId should be used across
the board.
2021-10-11 16:00:19 -04:00
Mike Gerwitz 7873d46afb tamer: Replace all &'static str in errors with SymbolId
Now that SymbolId implements Display and resolves, this works out well.
2021-10-11 15:39:53 -04:00
Mike Gerwitz 7e9271e189 tamer: span: Primitive Display impl
This outputs enough information to be a little bit useful in the event of an
error.  In the future, we'll want to provide a (likely non-Display)
implementation that provides line number and source file context with
the problem characters indicated, like Rust.
2021-10-11 14:14:43 -04:00
Mike Gerwitz a9140730d9 tamer: sym: Implement Display for SymbolId
This is a significant departure from my original plans---this makes it
_easy_ to display symbol values, despite me not wanting that to occur unless
absolutely necessary.

The reality is, based on the design of the system, they will only occur in
these situations:

  1. Writing to files;
  2. Displaying errors;
  3. Tests; or
  4. People not following the design of the system.

The fourth one is the most risky as people begin to contribute in the
future, but the reality is that those can be fixed as they are encountered,
since if they're not showing up in a profiler, then they must not be causing
much of a problem.
2021-10-11 13:52:35 -04:00
Mike Gerwitz 85909f1590 tamer: sym::SymbolStr: Remove
This removes `SymbolStr` in favor of, simply, `&'static str`.

The abstraction provided no additional safety since the slice was trivially
extracted (and commonly, in practice), and was inconvenient to work with.

This is part of a process of relaxing lookups so that symbols can be
conveniently displayed in errors; rather than trying to prevent the
developer from doing something bad, we'll just rely on conventions, hope
that it doesn't happen, and if it does, address it either at that time or
when it shows up in the profiler.
2021-10-11 12:58:48 -04:00
Mike Gerwitz 68397f1413 tamer: ir::xir: Add missing docs for QName, Prefix, LocalName
The docs still need to be improved, but they can be touched as we go.

This concludes the initial development of XIR.  That was much more involved
that I had originally intended, but the result is good.

DEV-10561
2021-10-11 11:56:03 -04:00
Mike Gerwitz bc5091d2a7 tamer: ir::xir (newtype_symbol!): Remove for now
This does not belong here and was more of a POC at the time.  It can be
added later on when I have the time; I have to move on.
2021-10-11 11:51:51 -04:00
Mike Gerwitz f65ec818ab tamer: obj::xmle::xir: Correct doc typos Xml{e=>}Writer 2021-10-11 11:51:32 -04:00
Mike Gerwitz 3e385d1a1b tamer: obj::xmle::xir: Finalize docs
This could be improved upon, but there will be more work coming up for this
to finalize Sections.

DEV-10561
2021-10-11 11:43:49 -04:00
Mike Gerwitz bc5e8ebe75 tamer: obj::xmle::xir: Extract ElemWrap into ir::xir::iter
This generalizes it a bit and provides tests, which was always the intent;
the existing code was POC to determine if this could be done without
performance degradation (see that commit for more information).
2021-10-11 10:33:24 -04:00
Mike Gerwitz cde08b125c tamer: span (DUMMY_SPAN): New constant
Rather than having to use lazy_static! in all these tests, we can derive an
unlimited number of dummy spans from this one using e.g. `offset_add`.
2021-10-11 10:29:58 -04:00
Mike Gerwitz cf239531e0 tamer: span (offset_add): New method
More will come in the future, including the ability to add two spans.
2021-10-11 10:28:47 -04:00
Mike Gerwitz de3d7ef393 tamer: span: Introduce twospan
The intent is to support the composition and decomposition of spans such
that (A, B) is as documented here.  This only performs the trivial case for
the sake of providing a convenient API when the developer would otherwise
just type (S, S).
2021-10-11 09:56:48 -04:00
Mike Gerwitz 1a2f6bd209 tamer: obj::xmle::xir: Extract ElemWrap into ir::xir::iter 2021-10-11 09:34:17 -04:00
Mike Gerwitz de62a2acbc tamer: ir::asg::section: Reduce fields
This is intended to represent the sections written to the final xmle file,
and there was unnecessary complexity in separating everything.

By reducing this IR further, we can begin to constrain its types to
eliminate some of the runtime panics and error checking we have/had in the
writer.
2021-10-11 09:07:48 -04:00
Mike Gerwitz f70f5653b2 tamer: ir::asg::section: Head and tail can have only one object
This is the beginning of a refactoring to simplify this implementation a
little bit.
2021-10-09 00:27:03 -04:00
Mike Gerwitz 0626629cb3 tamer: Remove old xmle writer and wip-xir-xmle-writer flag
The new writer has reached parity of the old, with the exception of some
edge case explicit error handling that should never occur (which will be
added), and cleanup/docs.

Removing this flag now allows me to perform that cleanup without having to
worry about updating the now-old implementation.

I ran `tameld` with the new writer against our production system with
numerous programs and a significant number of test cases, and diff'd the old
and new xmle files, and everything looks good.
2021-10-08 22:04:42 -04:00
Mike Gerwitz 82727a5d66 tamer: obj::xmle::xir::header: Remove Rust 2018 comment
We're on 2021 now.
2021-10-08 21:43:28 -04:00
Mike Gerwitz d616d9475c tamer: obj::xmle::xir: Complete writer functionality
This is a significant milestone, in the sense that it is the culmination of
the past month or so of work to prove that an Iterator-based XIR will be
viable for the system.

This barely had any impact on the performance from the previous commit
reporting the profiling.  This performs at least as well as the quick-xml
based writer.  In isolated benchmarks, it performs better, but in the real
world, the linker spends most of its time reading xmlo files, and so minor
differences in writing do not have a significant overall impact.

With that said, a lot of cleanup and documentation is still needed.  That is
the subject of the upcoming commits, before this writer can finalized.
2021-10-08 16:37:46 -04:00
Mike Gerwitz 929a6c9815 tamer: obj::xmle::xir::tree: Parse Text into Element
This simply adds support for Text nodes as a child of Element.  This support
unit tests for the upcoming change for xmle fragments.
2021-10-08 16:16:33 -04:00
Mike Gerwitz f0f6f89745 tamer: Makefile.am (bench-build): New target, default for all
Build the benchmarks by default to catch breakages without having to incur
the cost of actually running them.
2021-10-08 09:27:56 -04:00
Mike Gerwitz 75d2ecf4dd tamer: obj::xmle::xir: Consideration of simplified iterators
The previous iterators had to be used in a certain order because they mixed
concerns, out of concern for performance.  This attempts to chain even more
iterators to see how it may perform.

To be clear: this will be cleaned up.  This was just an experiment.

Here were profiles on the average of 50 runs of linking our largest program:

  Baseline, pre-XIR (with fragments removed from output)               0.8082
  XIR writer, pre-ElemWrap, no #[inline]                               0.7844s
  XIR writer, ElemWrap, no #[inline]                                   0.7918s
  XIR writer, ElemWrap, inlines in obj::xmle::xir                      0.7892s
  XIR writer, ElemWrap, inlines in obj::xmle::xir and ir::asg::section 0.7858s
  XIR writer, ElemWrap, inline in only ir::asg::section                0.781s
  Pre-ElemWrap, inlines in ir::asg::section                            0.7772s

These profiles are difficult, because they hit the filesystem so much.  I
write to /dev/null, but it reads 100s of xmlo files from disk.

It's clear that the impact is fairly modest and within a margin of error; as
such, I will continue down the path of writing code that's easier to grok
and maintain, since not doing so would be a micro-optimization relative to
the concerns of the rest of the system at this point.

But the purpose of all of this work was to determine whether an
iterator-based XIR would be viable.  It seems to be competitive.  I'll
finish up the writer reimplementation and move on.
2021-10-07 16:48:58 -04:00
Mike Gerwitz 2821098b40 .gitlab-ci.yml: Skip main build after stage build
Two reasons for this:

  1. It's unnecessary, since it's the same ref, so long as we actually build
     everything as part of the stage job; and
  2. In our environment, the token used doesn't have access to pull from the
     registry.

Fixing the latter item can be done at another time.
2021-10-07 15:55:22 -04:00
Mike Gerwitz f0ca3e88b7 .gitlab-ci.yml: Remove unnecessary SSH auth
We're using token-based auth now, using a project access token.
2021-10-07 13:25:55 -04:00
Mike Gerwitz 072a501ed5 .gitlab-ci.yml: Merge to main after successful stage pipeline 2021-10-07 13:09:40 -04:00
Mike Gerwitz 7f5064c665 tamer: obj::xmle::xir: Write l:map-from
This contains some awkward coupling for opening and closing tags to reduce
the complexity of the `Iterator` types that must be manually
specified.  That may be addressed shortly.
2021-10-05 16:13:47 -04:00
Austin Schaffer d54ef62a0d Fix import ordering 2021-10-04 17:15:02 -04:00
Mike Gerwitz 1a44e04333 tamer: ld: Write is unused outside of flag 2021-10-04 16:34:25 -04:00
Mike Gerwitz e2c9944f1b tamer: Move Sections map from unique from writer into Sections
We're implementing an new XIR-based writer and don't want to have to
duplicate this; it didn't really belong there to begin with.
2021-10-04 16:31:30 -04:00
Mike Gerwitz 004f5dc312 tamer: Read only a single map preproc:from from xmlo files
This was creating a heap-allocated `Vec` for each map symbol despite not
actually needing it.  We do have multiple froms for return map values.

But by the time we may want this type of thing, we'll have a different IR
for it anyway.
2021-10-04 14:59:33 -04:00
Mike Gerwitz 772619f6f0 tamer: Replace explicit array::IntoIter::new with IntoIter
Now that we're on 2021 Edition, the default behavior has changed to be
consistent.
2021-10-02 01:03:19 -04:00
Mike Gerwitz f9c9c95516 tamer: sym::prefill: Static symbol polymorphism
See the docs for a much deeper discussion.  In summary: traits do not
support static methods, and this is the workaround, which relies on unstable
nightly constant function features.

This implementation is tested using `qname_const!`, and will be utilized
with a new static type in a following commit.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 9d87962e96 tamer: Use Rust 2021 Edition
This will be stable Oct 21; this uses nightly for now.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 885d5e4d8f tamer: Switch back to nightly toolchain
This is to support two things:
  1. Early switch to 2021 Edition, which is stable Oct 21; and
  2. To make use of unstable const features.

The rationale is that switching to nightly does not really have any
significant downside for us, given that TAMER is used only by us and
the only risk is that unstable features may change a bit, which can be
mitigated with certain precautions.

The rationale for each unstable feature will be documented as they are used,
including documentation on what would be required to remove it and what
functionality would be lost / need to change in doing so.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 7c61a92d30 tamer: obj::xmle::xir: Minor clean and docs
This is far from fully documented; it's just a start.  I'll document fully
once the implementation is done, to ensure I don't waste time documenting
things that may change.
2021-10-02 00:58:14 -04:00
Mike Gerwitz 42188e80e7 tamer: obj::xmle::xir::test: Extract into own file
These are getting large and messy.

And I now notice that I never completed the header test after
prototyping.  Shame on me.

Also, errata from the previous commit message: the diffs are identical
_except for attribute escaping_ that is unnecessary; we're outputting data
read directly from existing XML files (output by Saxon), so characters are
already escaped as needed.

DEV-10561
2021-10-02 00:58:13 -04:00
Mike Gerwitz 7269e68b00 tamer: obj::xmle::xir: Complete l:dep
The `l:dep` section of the `xmle` file, after formatting (since XIR writes
without newlines and indentation), is now identical to the existing xmle
writer.  I can now move on to the other sections.

Note that the attribute movement in this commit is simply to get the diff to
properly align.  Once the current xmle writer is removed, I'll organize them
a bit more sensibly.

`obj::xmle::xir` also needs documentation, now that it's shown to be viable.
2021-09-30 13:06:30 -04:00
Mike Gerwitz acf55fad81 tamer: Intern desc from xmle on read
The new xmle writer was having to intern before write, which did not make
sense.

This continues with consistently using symbols throughout the system, and
is a smaller size than `String` as a bonus.
2021-09-29 23:31:07 -04:00
Mike Gerwitz 5250571f15 tamer: ir::asg::ident: Use symbols in place of string slice mapping
`IdentKind` needs to be written to `xmle` files and displayed in error
messages.  String slices were used when quick-xml was used for writing,
which will be going away with the new writer.
2021-09-29 23:18:23 -04:00
Mike Gerwitz fa4181770f tamer: src::ir::asg::ident::Dim: Assert n<10
This replaces a TODO with an assertion.
2021-09-29 16:26:41 -04:00
Mike Gerwitz 6864fbc1cd tamer: Start of XIR-based xmle writer
This has been a long time coming, and has been repeatedly stashed as other
parts of the system have evolved to support it.  The introduction of the XIR
tree was to write tests for this (which are sloppy atm).

This currently writes out the `xmle` header and _most_ of the `l:dep`
section; it's missing the object-type-specific attributes.  There is,
relatively speaking, not much more work to do here.

The feature flag `wip-xir-xmle-writer` was introduced to toggle this system
in place of `XmleWriter`.  Initial benchmarks show that it will be
competitive with the quick-xml-based writer, but remember that is not the
goal: the purpose of this is to test XIR in a production system before we
continue to implement it for a frontend, and to refactor so that we do not
have multiple implementations writing XML files (once we echo the source XML
files).

I'm excited to get this done with so that I can move on.  This has been
rather exhausting.
2021-09-28 14:52:53 -04:00
Mike Gerwitz 863d990cbd tamer: sym: 16-bit static symbol prefill
The 16-bit interner at present will be used only for span contexts.  In the
future, this interner may become specialized specifically for that, but for
now let's just re-use what we already have so that I can move on.

DEV-10733
2021-09-28 10:39:46 -04:00
Mike Gerwitz 96b16c6de9 tamer: sym::prefill::test::global_sanity_check: Note duplicate strings
I want to make it clear in the assertion that the problem could be caused by
duplicate strings.  We do not sort by string, because in part we may in the
future want to group certain symbols together in some arbitrary way so we
can compare ranges (using the markers).

If that doesn't end up happening, it may be better to just sort by string
to obviate the problem.
2021-09-24 16:25:29 -04:00
Mike Gerwitz db8a098452 tamer: sym: Minor documentation refinement
Mostly rewording.
2021-09-24 10:11:19 -04:00
Mike Gerwitz c71d36b154 tamer: sym::prefill: All-caps constants for static symbols
It's really awkward not having them caps, when not only are constants
expected to be, but also that we cannot maintain consistency between the
string and the identifier name in even the simplest of cases.

(We could use `r#`, but that's too cumbersome.)
2021-09-23 23:48:28 -04:00
Mike Gerwitz 785ca0fe9e tamer: sym::prefill: Remove StaticSymbolId in favor of refined types
`StaticSymbolId` was created before the more specific types, which render it
unnecessary.  If we need a generic type, it can be re-introduced, but using
`static_symbol_newtypes!`.
2021-09-23 23:35:45 -04:00