employer
/
tame
1
0
Fork 0
Code Packages Releases Activity
tame/tamer/Cargo.toml

62 lines
2.0 KiB
TOML
Raw Normal View History

TAMER: Initial commit
2019-11-14 16:43:07 -05:00
[package]
name = "tamer"
version = "0.0.0"
authors = ["Mike Gerwitz <mike.gerwitz@ryansg.com>"]
Cargo.toml: Tame {on=>in} Rust Changed to match README.md. This makes more sense too.
2019-11-27 13:47:42 -05:00
description="TAME in Rust"
TAMER: Initial commit
2019-11-14 16:43:07 -05:00
license="GPLv3+"
tamer: Use Rust 2021 Edition This will be stable Oct 21; this uses nightly for now.
2021-10-01 09:42:19 -04:00
edition = "2021"
TAMER: Initial commit
2019-11-14 16:43:07 -05:00
Introduce CARGO_BUILD_FLAGS This is intended to permit passing `--release`, since dev builds are terribly slow (e.g. 6s -> 0.2s). See README.md for more information.
2019-11-27 09:17:27 -05:00
[profile.dev]
# Release-level optimizations. Spending the extra couple of moments
# compile-time is well worth the huge savings we get at runtime. Note that
# this is still every so slightly slower than a release build; see other
# profile options for release at
# <https://doc.rust-lang.org/cargo/reference/manifest.html>.
opt-level = 3
TAMER: Initial commit
2019-11-14 16:43:07 -05:00
Cargo.toml (profile.release)[lto]: Enable
2019-12-03 12:17:44 -05:00
[profile.release]
lto = true
Makefile.am (bench): New target The configure script will determine if nightly is required for running benchmarks, because `test` is currently an unstable feature.
2019-12-04 09:57:08 -05:00
[profile.bench]
# We want our benchmarks to be representative of how well TAME will perform
# in a release.
lto = true
[DEV-7081] Add options to tameld We want to add an option to set the output file to the linker so we do not need to redirect output to awk any longer. This also adds integration tests for tameld.
2020-03-04 15:31:20 -05:00
[dev-dependencies]
assert_cmd = "0.10"
predicates = "1"
TAMER: Initial commit
2019-11-14 16:43:07 -05:00
[dependencies]
tamer: Start of XIR-based xmle writer This has been a long time coming, and has been repeatedly stashed as other parts of the system have evolved to support it. The introduction of the XIR tree was to write tests for this (which are sloppy atm). This currently writes out the `xmle` header and _most_ of the `l:dep` section; it's missing the object-type-specific attributes. There is, relatively speaking, not much more work to do here. The feature flag `wip-xir-xmle-writer` was introduced to toggle this system in place of `XmleWriter`. Initial benchmarks show that it will be competitive with the quick-xml-based writer, but remember that is not the goal: the purpose of this is to test XIR in a production system before we continue to implement it for a frontend, and to refactor so that we do not have multiple implementations writing XML files (once we echo the source XML files). I'm excited to get this done with so that I can move on. This has been rather exhausting.
2021-09-28 14:52:31 -04:00
arrayvec = ">= 0.7.1"
TAMER: Arena-based string interner Contrary to what I said previously, this replaces the previous implementation with an arena-backed internment system. The motivation for this change was investigating how Rustc performed its string interning, and why they chose to associate integer identifiers with symbols. The intent was originally to use Rustc's arena allocator directly, but that create pulled in far too many dependencies and depended on nightly Rust. Bumpalo provides a very similar implementation to Rustc's DroplessArena, so I went with that instead. Rustc also relies on a global, singleton interner. I do not do that here. Instead, the returned Symbol carries a lifetime of the underlying arena, as well as a pointer to the interned string. Now that this is put to rest, it's time to move on.
2019-12-23 23:26:42 -05:00
bumpalo = ">= 2.6.0"
tamer::sym: FNV => Fx Hash For strings of any notable length, Fx Hash outperforms FNV. Rustc also moved to this hash function and noticed performance improvements. Fortunately, as was accounted for in the design, this was a trivial switch. Here are some benchmarks to back up that claim: test hash_set::fnv::with_all_new_1000 ... bench: 133,096 ns/iter (+/- 1,430) test hash_set::fnv::with_all_new_1000_with_capacity ... bench: 82,591 ns/iter (+/- 592) test hash_set::fnv::with_all_new_rc_str_1000_baseline ... bench: 162,073 ns/iter (+/- 1,277) test hash_set::fnv::with_one_new_1000 ... bench: 37,334 ns/iter (+/- 256) test hash_set::fnv::with_one_new_rc_str_1000_baseline ... bench: 18,263 ns/iter (+/- 261) test hash_set::fx::with_all_new_1000 ... bench: 85,217 ns/iter (+/- 1,111) test hash_set::fx::with_all_new_1000_with_capacity ... bench: 59,383 ns/iter (+/- 752) test hash_set::fx::with_all_new_rc_str_1000_baseline ... bench: 98,802 ns/iter (+/- 1,117) test hash_set::fx::with_one_new_1000 ... bench: 42,484 ns/iter (+/- 1,239) test hash_set::fx::with_one_new_rc_str_1000_baseline ... bench: 15,000 ns/iter (+/- 233) test hash_set::with_all_new_1000 ... bench: 137,645 ns/iter (+/- 1,186) test hash_set::with_all_new_rc_str_1000_baseline ... bench: 163,129 ns/iter (+/- 1,725) test hash_set::with_one_new_1000 ... bench: 59,051 ns/iter (+/- 1,202) test hash_set::with_one_new_rc_str_1000_baseline ... bench: 37,986 ns/iter (+/- 771)
2019-12-10 15:32:25 -05:00
fxhash = ">= 0.2.1"
tamer: Remove default SymbolIndex (et al) index type Oh boy. What a mess of a change. This demonstrates some significant issues we have with Symbol. I had originally modelled the system a bit after Rustc's, but deviated in certain regards: 1. This has a confurable base type to enable better packing without bit twiddling and potentially unsafe tricks I'd rather avoid unless necessary; and 2. The lifetime is not static, and there is no global, singleton interner; and 3. I pass around references to a Symbol rather than passing around an index into an interner. For #3---this is done because there's no singleton interner and therefore resolving a symbol requires a direct reference to an available interner. It also wasn't clear to me (and still isn't, in fact) whether more than one interner may be used for different contexts. But, that doesn't preclude removing lifetimes and just passing around indexes; in fact, I plan to do this in the frontend where the parser and such will have direct interner access and can therefore just look up based on a symbol index. We could reserve references for situations where exposing an interner would be undesirable. Anyway, more to come...
2021-07-29 14:26:40 -04:00
petgraph = "0.5.1" # TODO: petgraph-graphml holds this back
tamer: xir::XirString: WIP implementation (likely going away) I'm not fond of this implementation, which is why it's not fully completed. I wanted to commit this for future reference, and take the opportunity to explain why I don't like it. First: this task started as an idea to implement a third variant to AttrValue and friends that indicates that a value is fixed, in the sense of a fixed-point function: escaped or unescaped, its value is the same. This would allow us to skip wasteful escape/unescape operations. In doing so, it became obvious that there's no need to leak this information through the API, and indeed, no part of the system should care. When we read XML, it should be unescaped, and when we write, it should be escaped. The reason that this didn't quite happen to begin with was an optimization: I'll be creating an echo writer in place of the current filesystem-based copy in tamec shortly, and this would allow streaming XIR directly from the reader to the writer without any unescaping or re-escaping. When we unescape, we know the value that it came from, so we could simply store both symbols---they're 32-bit, so it results in a nicely compressed 64-bit value, so it's essentially cost-free, as long as we accept the expense of internment. This is `XirString`. Then, when we want to escape or unescape, we first check to see whether a symbol already exists and, if so, use it. While this works well for echoing streams, it won't work all that well in practice: the unescaped SymbolId will be taken and the XirString discarded, since nothing after XIR should be coupled with it. Then, when we later construct a XIR stream for writting, XirString will no longer be available and our previously known escape is lost, so the writer will have to re-escape. Further, if we look at XirString's generic for the XirStringEscaper---it uses phantom, which hints that maybe it's not in the best place. Indeed, I've already acknowledged that only a reader unescapes and only a writer escapes, and that the rest of the system works with normal (unescaped) values, so only readers and writers should be part of this process. I also already acknowledged that XirString would be lost and only the unescaped SymbolId would be used. So what's the point of XirString, then, if it won't be a useful optimization beyond the temporary echo writer? Instead, we can take the XirStringWriter and implement two caches on that: mapping SymbolId from escaped->unescaped and vice-versa. These can be simple vectors, since SymbolId is a 32-bit value we will not have much wasted space for symbols that never get read or written. We could even optimize for preinterned symbols using markers, though I'll probably not do so, and I'll explain why later. If we do _that_, we get even _better_ optimizations through caching that _will_ apply in the general case (so, not just for echo), and we're able to ditch XirString entirely and simply use a SymbolId. This makes for a much more friendly API that isn't leaking implementation details, though it _does_ put an onus on the caller to pass the encoder to both the reader and the writer, _if_ it wants to take advantage of a cache. But that burden is not significant (and is, again, optional if we don't want it). So, that'll be the next step.
2021-11-10 09:42:18 -05:00
quick-xml = ">= 0.23.0-alpha3"
[DEV-7081] Add options to tameld We want to add an option to set the output file to the linker so we do not need to redirect output to awk any longer. This also adds integration tests for tameld.
2020-03-04 15:31:20 -05:00
getopts = "0.2"
exitcode = "1.1.2"
Revert "tamer: Cargo.toml (dependencies)[lazy_static]: Remove (now used)" This reverts commit 4fd6313cd22737c0e31d8d79f9c2087fdc51bd80. ...and now I need it for tests.
2021-08-12 16:08:34 -04:00
lazy_static = ">= 1.4.0"
[DEV-7504] Add GraphML generation We want to be able to build a representation of the dependency graph so we can easily inspect it. We do not want to make GraphML by default. It is better to use a tool. We use "petgraph-graphml".
2020-04-30 14:33:10 -04:00
petgraph-graphml = ">= 2.0.1"
tamer: Remove default SymbolIndex (et al) index type Oh boy. What a mess of a change. This demonstrates some significant issues we have with Symbol. I had originally modelled the system a bit after Rustc's, but deviated in certain regards: 1. This has a confurable base type to enable better packing without bit twiddling and potentially unsafe tricks I'd rather avoid unless necessary; and 2. The lifetime is not static, and there is no global, singleton interner; and 3. I pass around references to a Symbol rather than passing around an index into an interner. For #3---this is done because there's no singleton interner and therefore resolving a symbol requires a direct reference to an available interner. It also wasn't clear to me (and still isn't, in fact) whether more than one interner may be used for different contexts. But, that doesn't preclude removing lifetimes and just passing around indexes; in fact, I plan to do this in the frontend where the parser and such will have direct interner access and can therefore just look up based on a symbol index. We could reserve references for situations where exposing an interner would be undesirable. Anyway, more to come...
2021-07-29 14:26:40 -04:00
static_assertions = ">= 1.1.0"
tamer: memchr benches This adds benchmarking for the memchr crate. It is used primarily by quick-xml at the moment, but the question is whether to rely on it for certain operations for XIR. The benchmarking on an Intel Xeon system shows that memchr and Rust's contains() perform very similarly on small inputs, matching against a single character, and so Rust's built-in should be preferred in that case so that we're using APIs that are familiar to most people. When larger inputs are compared against, there's a greater benefit (a little under ~2x). When comparing against two characters, they are again very close. But look at when we compare two characters against _multiple_ inputs: running 24 tests test large_str::one::memchr_early_match ... bench: 4,938 ns/iter (+/- 124) test large_str::one::memchr_late_match ... bench: 81,807 ns/iter (+/- 1,153) test large_str::one::memchr_non_match ... bench: 82,074 ns/iter (+/- 1,062) test large_str::one::rust_contains_one_byte_early_match ... bench: 9,425 ns/iter (+/- 167) test large_str::one::rust_contains_one_byte_late_match ... bench: 123,685 ns/iter (+/- 3,728) test large_str::one::rust_contains_one_byte_non_match ... bench: 123,117 ns/iter (+/- 2,200) test large_str::one::rust_contains_one_char_early_match ... bench: 9,561 ns/iter (+/- 507) test large_str::one::rust_contains_one_char_late_match ... bench: 123,929 ns/iter (+/- 2,377) test large_str::one::rust_contains_one_char_non_match ... bench: 122,989 ns/iter (+/- 2,788) test large_str::two::memchr2_early_match ... bench: 5,704 ns/iter (+/- 91) test large_str::two::memchr2_late_match ... bench: 89,194 ns/iter (+/- 8,546) test large_str::two::memchr2_non_match ... bench: 85,649 ns/iter (+/- 3,879) test large_str::two::rust_contains_two_char_early_match ... bench: 66,785 ns/iter (+/- 3,385) test large_str::two::rust_contains_two_char_late_match ... bench: 2,148,064 ns/iter (+/- 21,812) test large_str::two::rust_contains_two_char_non_match ... bench: 2,322,082 ns/iter (+/- 22,947) test small_str::one::memchr_mid_match ... bench: 4,737 ns/iter (+/- 842) test small_str::one::memchr_non_match ... bench: 5,160 ns/iter (+/- 62) test small_str::one::rust_contains_one_byte_non_match ... bench: 3,930 ns/iter (+/- 35) test small_str::one::rust_contains_one_char_mid_match ... bench: 3,677 ns/iter (+/- 618) test small_str::one::rust_contains_one_char_non_match ... bench: 5,415 ns/iter (+/- 221) test small_str::two::memchr2_mid_match ... bench: 5,488 ns/iter (+/- 888) test small_str::two::memchr2_non_match ... bench: 6,788 ns/iter (+/- 134) test small_str::two::rust_contains_two_char_mid_match ... bench: 6,203 ns/iter (+/- 170) test small_str::two::rust_contains_two_char_non_match ... bench: 7,853 ns/iter (+/- 713) Yikes. With that said, we won't be comparing against such large inputs short-term. The larger strings (fragments) are copied verbatim, and not compared against---but they _were_ prior to the previous commit that stopped unencoding and re-encoding. So: Rust built-ins for inputs that are expected to be small.
2021-08-18 14:18:24 -04:00
memchr = ">= 2.3.4" # quick-xml expects =2.3.4 at the time
tamer: ir::asg::ident: AsRef impls for SymbolId types This commit will make more sense once the broader context is committed, but it's needed for lowering from `Sections` into a XIR stream. This will also change once we pre-allocate symbols, like rustc, when the interner is initialized. This is my first use of the `paste` crate, which is used to generate identifiers. So this is partly an experiment, and it seems much better than having to write a proc macro, at least at this point in time. If this code stays around, it'll probably be generalized further and used elsewhere, but I'd prefer not to go this route long-term.
2021-09-20 16:46:16 -04:00
paste = ">= 1.0.5"
[DEV-7081] Add options to tameld We want to add an option to set the output file to the linker so we do not need to redirect output to awk any longer. This also adds integration tests for tameld.
2020-03-04 15:31:20 -05:00
tamer: Initial frontend concept This introduces the beginnings of frontends for TAMER, gated behind a `wip-features` flag. This will be introduced in stages: 1. Replace the existing copy with a parser-based copy (echo back out the tokens), when the flag is on. 2. Begin to parse portions of the source, augmenting the output xmlo (xmli at the moment). The XSLT-based compiler will be modified to skip compilation steps as necessary. As portions of the compilation are implemented in TAMER, they'll be placed behind their own feature flags and stabalized, which will incrementally remove the compilation steps from the XSLT-based system. The result should be substantial incremental performance improvements. Short-term, the priorities are for loading identifiers into an IR are (though the order may change): 1. Echo 2. Imports 3. Extern declarations. 4. Simple identifiers (e.g. param, const, template, etc). 5. Classifications. 6. Documentation expressions. 7. Calculation expressions. 8. Template applications. 9. Template definitions. 10. Inline templates. After each of those are done, the resulting xmlo (xmli) will have fully reconstructed the source document from the IR produced during parsing.
2021-07-23 22:24:08 -04:00
# Feature flags can be specified using `./configure FEATURES=foo,bar,baz`.
#
# Flags beginning with "wip-" are short-lived flags that exist only during
# development of a particular feature; you should not hard-code them
# anywhere, since the build will break once they are removed. Enabling WIP
# flags should also be expected to cause undesirable behavior in some form
# or another. Once WIP features are finalized, they are enabled by default
# and the flag removed.
[features]
# Process source files using available frontends rather than copying
# the files verbatim to XMLI files. This begins the process of moving
# compilation from XSLT into TAMER, and so the XSLT-based compiler must be
# expecting it so that it can skip those compilation steps.
wip-frontends = []
tamer: Start of XIR-based xmle writer This has been a long time coming, and has been repeatedly stashed as other parts of the system have evolved to support it. The introduction of the XIR tree was to write tests for this (which are sloppy atm). This currently writes out the `xmle` header and _most_ of the `l:dep` section; it's missing the object-type-specific attributes. There is, relatively speaking, not much more work to do here. The feature flag `wip-xir-xmle-writer` was introduced to toggle this system in place of `XmleWriter`. Initial benchmarks show that it will be competitive with the quick-xml-based writer, but remember that is not the goal: the purpose of this is to test XIR in a production system before we continue to implement it for a frontend, and to refactor so that we do not have multiple implementations writing XML files (once we echo the source XML files). I'm excited to get this done with so that I can move on. This has been rather exhausting.
2021-09-28 14:52:31 -04:00
tamer: wip-xml-xir-reader flag and setup The original plan was to modify the existing reader to use the new XmlXirReader, but that's going to be a lot of ongoing uncommitted work, with both tests and implementation. The better option seems to be to reimplement it, since so many things are changing. This flag will be short-lived and removed as soon as the implementation is complete. DEV-10863
2021-10-25 12:02:46 -04:00
# Use the XIR-based reader for parsing xmlo files in tameld. This flag will
# exist only while reimplementing the XmloReader.
wip-xmlo-xir-reader = []