Commit Graph

556 Commits (ab3aec980d2e3cf55d01b6058392296c884f9a49)

Author SHA1 Message Date
Mike Gerwitz ab3aec980d TAMER: POC: Use FxHash to remove nondeterminism
The default SipHash is a cryptographic hash and causes ordering to change
between runs.
2020-02-26 10:49:00 -05:00
Mike Gerwitz 645908e258 TAMER: xmle output changes to support Summary Page
Co-Authored-By: Joseph Frazer <joseph.frazer@ryansg.com>
2020-02-26 10:49:00 -05:00
Mike Gerwitz 6939753ca0 TAMER: POC: Output xmle
This is a working proof-of-concept that will be finalized in future commits.
2020-02-26 10:49:00 -05:00
Mike Gerwitz 85a4934db5 TAMER: Symbol source data and metadata 2020-02-26 10:49:00 -05:00
Mike Gerwitz bcc2ab1221 TAMER: Initial abstract semantic graph (ASG)
This begins to introduce the ASG, backed by Petgraph.  The API will continue
to evolve, and Petgraph will likely be encapsulated so that our
implementation can vary independently from it (or even remove it in the
future).
2020-02-26 10:48:59 -05:00
Mike Gerwitz f177b6ae5d configure.ac: Rust 1.{39>41}.0 version bump
Relaxes orphan rules for foreign traits.

This also modifies the error to suggest how to update using rustup.
2020-02-25 16:46:28 -05:00
Mike Gerwitz 10b9caa7ad TAMER: Fail on empty fragment ids (and fix underlying problem) 2020-02-25 16:46:28 -05:00
Mike Gerwitz a0893da577 TAMER: xmlo: Add Package event 2020-02-25 16:46:27 -05:00
Mike Gerwitz a8726918f7 TAMER: poc: Use xmlo reader
TODO: More information
2020-02-25 16:46:27 -05:00
Mike Gerwitz a929c8cae4 TAMER: xmlo reader
This introduces the reader for xmlo files produced by the XSLT-based
compiler.  It is an initial implementation but is not complete; see future
commits.
2020-02-25 16:46:25 -05:00
Mike Gerwitz db52fcdb30 Makefile.am (html-am): Add --document-private-items
This generated documenation is only going to be read be developers,
and the private information is very useful to them.
2020-02-25 16:10:57 -05:00
Mike Gerwitz 6aae741162 TAMER (sym::Interner::intern_utf8_unchecked): New function
This removes boilerplate for reading xmlo files.  See next commit.
2020-02-25 16:10:55 -05:00
Mike Gerwitz e8cd378d59 TAMER: Display for Symbol
One of the benefits of storing a reference to the interned string on the
symbol itself is that we get to get its underlying value essentially for
free.
2020-02-24 14:56:28 -05:00
Mike Gerwitz ff0c8bb34f Order symtable, sym-dep, fragments
This ordering will simplify streaming processing of xmlo files in
TAMER.  Specifically, we know that symbols will have been declared by the
time dependencies are added to the graph (and so we should only be creating
edges to existing nodes); and we can halt reading as soon as the closing
fragments tag is encountered, avoiding parsing the entirety of these massive
XML files.

On one particularly large program, this cuts time down from ~0.333s to
~0.300 in the POC linker.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 1f4db84f24 TAMER: Arena-based string interner
Contrary to what I said previously, this replaces the previous
implementation with an arena-backed internment system.  The motivation for
this change was investigating how Rustc performed its string interning, and
why they chose to associate integer identifiers with symbols.

The intent was originally to use Rustc's arena allocator directly, but that
create pulled in far too many dependencies and depended on nightly
Rust.  Bumpalo provides a very similar implementation to Rustc's
DroplessArena, so I went with that instead.

Rustc also relies on a global, singleton interner.  I do not do that
here.  Instead, the returned Symbol carries a lifetime of the underlying
arena, as well as a pointer to the interned string.

Now that this is put to rest, it's time to move on.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 176d099fb6 tamer::sym: FNV => Fx Hash
For strings of any notable length, Fx Hash outperforms FNV.  Rustc also
moved to this hash function and noticed performance
improvements.  Fortunately, as was accounted for in the design, this was a
trivial switch.

Here are some benchmarks to back up that claim:

test hash_set::fnv::with_all_new_1000                 ... bench:     133,096 ns/iter (+/- 1,430)
test hash_set::fnv::with_all_new_1000_with_capacity   ... bench:      82,591 ns/iter (+/- 592)
test hash_set::fnv::with_all_new_rc_str_1000_baseline ... bench:     162,073 ns/iter (+/- 1,277)
test hash_set::fnv::with_one_new_1000                 ... bench:      37,334 ns/iter (+/- 256)
test hash_set::fnv::with_one_new_rc_str_1000_baseline ... bench:      18,263 ns/iter (+/- 261)
test hash_set::fx::with_all_new_1000                  ... bench:      85,217 ns/iter (+/- 1,111)
test hash_set::fx::with_all_new_1000_with_capacity    ... bench:      59,383 ns/iter (+/- 752)
test hash_set::fx::with_all_new_rc_str_1000_baseline  ... bench:      98,802 ns/iter (+/- 1,117)
test hash_set::fx::with_one_new_1000                  ... bench:      42,484 ns/iter (+/- 1,239)
test hash_set::fx::with_one_new_rc_str_1000_baseline  ... bench:      15,000 ns/iter (+/- 233)
test hash_set::with_all_new_1000                      ... bench:     137,645 ns/iter (+/- 1,186)
test hash_set::with_all_new_rc_str_1000_baseline      ... bench:     163,129 ns/iter (+/- 1,725)
test hash_set::with_one_new_1000                      ... bench:      59,051 ns/iter (+/- 1,202)
test hash_set::with_one_new_rc_str_1000_baseline      ... bench:      37,986 ns/iter (+/- 771)
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0d2bb5de59 Makefile.am (clean): New target
Not sure how I missed this one.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 541fbffc2e tameld: Move documentation to tamer::ld 2020-02-24 14:56:28 -05:00
Mike Gerwitz f2b24e6505 HashMapInterner: New interner, docs, and benchmarks
This interner will be suitable for providing an index to look up nodes in
the ASG.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 9a98644213 TAMER: sym::tests: Generate with macro
This will be used for generating the common tests between HashSet and
HashMap implementations.

This is my first macro in Rust.  There does not seem to be a way to
concatenate identifiers (!), so I'm placing them within modules
instead.  That ended up working out just fine, since then I can use a type
to provide the SUT.
2020-02-24 14:56:28 -05:00
Mike Gerwitz e4e0089815 TAMER: Initial string interning abstraction
This is missing two key things that I'll add shortly: a HashMap-based one
for use in the ASG for node mapping, and an entry-based system for
manipulations.

This has been a nice start for exploring various aspects of Rust
development, as well as conventions that I'd like to implement.  In
particular:

  - Robust documentation intended to guide people through learning the
    necessary material about the compiler, as well as related work to
    rationalize design decisions;
  - Benchmarks;
  - TDD;
  - And just getting used to Rust in general.

I've beat this one to death, so I'll commit this and make smaller changes
going forward to show how easily it can evolve.

(This module was originally named `intern` but this commit and those that
follow rewrote it to `sym`.)
2020-02-24 14:56:28 -05:00
Mike Gerwitz 593faa3491 Makefile.am (html-am): Run doc tests
Ensure that we have good examples before generating docs.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 3248c429fe Makefile.am (doc, html): Use intra_rustdoc_links
This is enabled by default in nightly, and is not available at all in
stable.  Considering the PITA that it will be to go back and rewrite docs to
use the new format, and how important of a feature this is, we will just
make use of it now.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0147cb7cb4 Makefile.am (bench): New target
The configure script will determine if nightly is required for running
benchmarks, because `test` is currently an unstable feature.
2020-02-24 14:56:28 -05:00
Mike Gerwitz 0acc21f16f Makefile.am (check): Check whether formatting is required
Given that developers should be doing TDD and therefore running this target
frequently, this has the effect of providing immediate feedback when
formatting is needed and outputting a diff.  Developers will then quickly
understand what changes need to be made to avoid future issues (and can run
`cargo fmt` to fix it), at which point they'll rarely ever encounter
formatting errors.

The original purpose was to ensure pipelines fail when the formatter has not
been run.
2020-02-24 14:56:28 -05:00
Austin Schaffer e1076ce388 [DEV-6306] Add retry template
[DEV-6306] Add testing instructions to README.md

[DEV-6306] Add assertion to retry template
2020-02-20 08:36:39 -05:00
Joseph Frazer 9f02781a5b [DEV-6947] Add template to match UI values
The UI values need to match AND the question needs to be
visible. We do not have the visibility classifications yet, so we need to
define externs to allow this to build.
2020-01-31 16:27:04 -05:00
Joseph Frazer f2cbc5f8ad [DEV-6947] Allow param values to remove underscores 2020-01-31 16:27:04 -05:00
Austin Schaffer 0db05c442c Pass canterm flag to raters 2020-01-29 11:14:13 -05:00
Mike Gerwitz bc28af0145 Reduce size of compiler output
This both reduces some of the output and permits it to be run through
Google's Closure compiler.  Combined, this has the potential to halve the
size of classification-heavy executables, like the UI's classifier.
2020-01-23 15:01:19 -05:00
Mike Gerwitz 696a5ab371 build-aux/closure-externs.js: New file 2020-01-22 16:31:57 -05:00
Mike Gerwitz 7a2ce00ed5 src/current/compiler/js.xsl: Remove inline defaults for anyValue
This not only reduces file size, but also has a significant performance
benefit for the UI, which is almost entirely classifications.  A run for one
of our systems was reduced from 1m30s to 11s from this change.
2020-01-22 16:31:16 -05:00
Mike Gerwitz 46d5ed286c src/current/compiler/js.xsl: Strip unused result-set (@yields alt) 2020-01-22 16:31:16 -05:00
Mike Gerwitz 661684f1e4 src/current/compiler/js.xsl: Remove last anyValue arg by default
This was used to provide additional information on the stack for debugging
the compiled code.  Since this is very rarely needed, and is only needed by
someone debugging the compiler, it can be manually enabled if desired.

This also wraps it so that it'll be stripped if it is included.
2020-01-22 16:31:16 -05:00
Mike Gerwitz b51f7fa042 src/current/compiler/js.xsl: {._CMATCH_=>[_CMATCH_]}
This was confusing Closure Compiler.
2020-01-22 16:31:16 -05:00
Mike Gerwitz 47d5dc238c src/current/compiler/js.xsl: @expose Closure Compiler annotations
This is deprecated, but neither of the recommended @export or @nocollapse
work the same way.
2020-01-22 16:31:09 -05:00
Mike Gerwitz 97806d5602 src/current/compiler/js.xsl: Remove dead arg check code
This was removed during The Great Refactoring.
It will be replaced with a better systemin TAMER.
2020-01-22 16:30:53 -05:00
Mike Gerwitz e0a78c2ed6 src/current/compiler/js-calc.xsl (compile-calc)[c:let]: Remove global assignment
The previous code was unintentionally assigning to an undefined global
variable.
2020-01-22 16:30:53 -05:00
Mike Gerwitz 0718d80257 bin/tamed: Fail without explicit DONE
We want to fail e.g. on a JVM crash.
2020-01-21 11:40:14 -05:00
Mike Gerwitz 61fe1af1cb build: Add revision files for xml{o,e}
This will force a rebuild and will be useful for upcoming changes.
2020-01-14 01:13:51 -05:00
Mike Gerwitz be296a241a build-aux/Makefile.am: Optional timestamping
Note that, because of the way this is implemented, the timestamps may become
mangled (multiple per line) for parallel builds.

Output can be prettied up in the future.
2020-01-02 10:42:08 -05:00
Mike Gerwitz 3cb67109ec Cargo.toml (profile.release)[lto]: Enable 2020-01-02 10:40:52 -05:00
Joseph Frazer 2a91d7680d [DEV-6595] Loosen the way we find classification matches
Merge branch 'jira-6595'

* jira-6595:
  [DEV-6595] Loosen the way we find classification matches
2019-12-10 09:42:11 -05:00
Joseph Frazer cdacd1d93d [DEV-6595] Loosen the way we find classification matches
The `<t:match-class-code-lookup />` matches were not showing in the
summary pages. I loosened the selector so it is able to find the matches
when it generates the summary pages.
2019-12-10 08:08:52 -05:00
Mike Gerwitz 2c1ff90d0a TAMER: tameld: Proof-of-concept
This is a POC playing around with Rust to demonstrate how the linker could
be approached and to gather benchmarks.
2019-12-02 15:21:46 -05:00
Mike Gerwitz 8455a38a1d Graph-based POC
This makes use of Petgraph for representing the dependency graph and uses a
separate data structure for both string interning and indexing by symbol
name.
2019-12-02 10:05:48 -05:00
Mike Gerwitz d78d81d721 Cargo.toml: Add petgraph
This will be used to represent the dependency graph.
2019-12-02 10:00:53 -05:00
Mike Gerwitz 717375a84a Cargo.toml: Tame {on=>in} Rust
Changed to match README.md.  This makes more sense too.
2019-12-02 10:00:53 -05:00
Mike Gerwitz 8374541965 tamer: Initial baisc POC with no XML output
This is garbage code.  Do not use it.  It is intentionally throwaway.

While I've researched Rust, I haven't actually _used_ it for a project, so
this is a combination of me exploring various ways of accomplishing the
problem and forcing myself to learn certain aspects of the language.

I'll likely be using petgraph, and this also currently lacks symbol
abstractions.  This commit also performs far too much heap allocation
copying strings around.  But it _does_ perform the topological sort.

Since this only stores the symbol name, it lacks enough information about
the symbol to perform a proper linking.
2019-12-02 10:00:53 -05:00
Mike Gerwitz e53482f2a3 Introduce CARGO_BUILD_FLAGS
This is intended to permit passing `--release`, since dev builds are
terribly slow (e.g. 6s -> 0.2s).  See README.md for more information.
2019-12-02 10:00:49 -05:00