employer
/
tame
1
0
Fork 0
Code Packages Releases Activity
tame/tamer/src/lib.rs

47 lines
1.2 KiB
Rust
Raw Normal View History

tamer: Initial baisc POC with no XML output This is garbage code. Do not use it. It is intentionally throwaway. While I've researched Rust, I haven't actually _used_ it for a project, so this is a combination of me exploring various ways of accomplishing the problem and forcing myself to learn certain aspects of the language. I'll likely be using petgraph, and this also currently lacks symbol abstractions. This commit also performs far too much heap allocation copying strings around. But it _does_ perform the topological sort. Since this only stores the symbol name, it lacks enough information about the symbol to perform a proper linking.
2019-11-27 09:18:17 -05:00
// TAME in Rust (TAMER)
//
Copyright year update 2021
2021-07-22 15:00:15 -04:00
// Copyright (C) 2014-2021 Ryan Specialty Group, LLC.
Copyright year 2020 update
2020-03-06 11:05:18 -05:00
//
// This file is part of TAME.
tamer: Initial baisc POC with no XML output This is garbage code. Do not use it. It is intentionally throwaway. While I've researched Rust, I haven't actually _used_ it for a project, so this is a combination of me exploring various ways of accomplishing the problem and forcing myself to learn certain aspects of the language. I'll likely be using petgraph, and this also currently lacks symbol abstractions. This commit also performs far too much heap allocation copying strings around. But it _does_ perform the topological sort. Since this only stores the symbol name, it lacks enough information about the symbol to perform a proper linking.
2019-11-27 09:18:17 -05:00
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
TAMER: Initial string interning abstraction This is missing two key things that I'll add shortly: a HashMap-based one for use in the ASG for node mapping, and an entry-based system for manipulations. This has been a nice start for exploring various aspects of Rust development, as well as conventions that I'd like to implement. In particular: - Robust documentation intended to guide people through learning the necessary material about the compiler, as well as related work to rationalize design decisions; - Benchmarks; - TDD; - And just getting used to Rust in general. I've beat this one to death, so I'll commit this and make smaller changes going forward to show how easily it can evolve. (This module was originally named `intern` but this commit and those that follow rewrote it to `sym`.)
2019-12-06 15:03:29 -05:00
//! An incremental rewrite of TAME in Rust.
tamer: Rust 1.{42=>48}.0 for stable intra-doc links without nightly
2021-06-21 13:10:00 -04:00
// We build docs for private items
#![allow(rustdoc::private_intra_doc_links)]
TAMER: Initial abstract semantic graph (ASG) This begins to introduce the ASG, backed by Petgraph. The API will continue to evolve, and Petgraph will likely be encapsulated so that our implementation can vary independently from it (or even remove it in the future).
2020-01-12 22:59:16 -05:00
pub mod global;
[DEV-7087] TAMER: symbol_dummy! macro
2020-03-24 14:14:05 -04:00
tamer: Remove default SymbolIndex (et al) index type Oh boy. What a mess of a change. This demonstrates some significant issues we have with Symbol. I had originally modelled the system a bit after Rustc's, but deviated in certain regards: 1. This has a confurable base type to enable better packing without bit twiddling and potentially unsafe tricks I'd rather avoid unless necessary; and 2. The lifetime is not static, and there is no global, singleton interner; and 3. I pass around references to a Symbol rather than passing around an index into an interner. For #3---this is done because there's no singleton interner and therefore resolving a symbol requires a direct reference to an available interner. It also wasn't clear to me (and still isn't, in fact) whether more than one interner may be used for different contexts. But, that doesn't preclude removing lifetimes and just passing around indexes; in fact, I plan to do this in the frontend where the parser and such will have direct interner access and can therefore just look up based on a symbol index. We could reserve references for situations where exposing an interner would be undesirable. Anyway, more to come...
2021-07-29 14:26:40 -04:00
#[macro_use]
extern crate static_assertions;
TAMER: sym::Interner::index_lookup This was originally omitted because there wasn't a use case for it. Now that we're adding context to errors, however, an owned value is highly desirable. This adds almost no measurable overhead to the internment system in benchmarks (largely within the margin of error).
2020-04-29 00:48:07 -04:00
tamer: XIR: Working concept This is a working streaming IR for XML. I want to get this committed before I go further cleaning it up and integrating it into the xmle writer. This is lacking detailed documentation, and the names of things may end up changing. Initial benchmarks do show that it has a ~2x performance improvement over quick-xml when dealing with two attributes on a node, and I suspect that improvement will increase with the number of attributes. We will see how it compares in real-world benchmarks once the linker has been modified to use it. The goal isn't to _avoid_ quick-xml---it'll be used in the future for things like escaping that would be a huge waste to implement ourselves. It just so happened that quick-xml was not beneficial for these changes; indeed, its own writer is fairly simple for the portions that were implemented here, so there's no use in fighting with its API, particularly around attributes and our need to explicitly control whitespace (with the intent of handling code formatters in the future). To put this into perspective: the reason this work is being done isn't to refactor the linker, or to speed it up, but to generalize XML writing and provide a suitable IR for use in the compiler. The first step of the frontend is to essentially echo the XML token stream back out so we can incrementally parse it and do something useful, to incrementally rewrite the compiler in Rust.
2021-08-20 10:09:55 -04:00
#[cfg(test)]
#[macro_use]
extern crate lazy_static;
tamer: Initial frontend concept This introduces the beginnings of frontends for TAMER, gated behind a `wip-features` flag. This will be introduced in stages: 1. Replace the existing copy with a parser-based copy (echo back out the tokens), when the flag is on. 2. Begin to parse portions of the source, augmenting the output xmlo (xmli at the moment). The XSLT-based compiler will be modified to skip compilation steps as necessary. As portions of the compilation are implemented in TAMER, they'll be placed behind their own feature flags and stabalized, which will incrementally remove the compilation steps from the XSLT-based system. The result should be substantial incremental performance improvements. Short-term, the priorities are for loading identifiers into an IR are (though the order may change): 1. Echo 2. Imports 3. Extern declarations. 4. Simple identifiers (e.g. param, const, template, etc). 5. Classifications. 6. Documentation expressions. 7. Calculation expressions. 8. Template applications. 9. Template definitions. 10. Inline templates. After each of those are done, the resulting xmlo (xmli) will have fully reconstructed the source document from the IR produced during parsing.
2021-07-23 22:24:08 -04:00
#[cfg(feature = "wip-frontends")]
pub mod frontend;
[DEV-7084] TAMER: fs: Basic filesystem abstraction This also includes an implementation to visit paths only once. Note that it does not yet canonicalize the path before visiting, so relative paths to the same file can slip through, and relative paths to _different_ files could be erroneously considered to have been visited. This will be fixed in an upcoming commit.
2020-04-06 16:13:32 -04:00
pub mod fs;
TAMER: xmlo reader This introduces the reader for xmlo files produced by the XSLT-based compiler. It is an initial implementation but is not complete; see future commits.
2020-01-09 10:55:55 -05:00
pub mod ir;
tamer: Initial baisc POC with no XML output This is garbage code. Do not use it. It is intentionally throwaway. While I've researched Rust, I haven't actually _used_ it for a project, so this is a combination of me exploring various ways of accomplishing the problem and forcing myself to learn certain aspects of the language. I'll likely be using petgraph, and this also currently lacks symbol abstractions. This commit also performs far too much heap allocation copying strings around. But it _does_ perform the topological sort. Since this only stores the symbol name, it lacks enough information about the symbol to perform a proper linking.
2019-11-27 09:18:17 -05:00
pub mod ld;
TAMER: xmlo reader This introduces the reader for xmlo files produced by the XSLT-based compiler. It is an initial implementation but is not complete; see future commits.
2020-01-09 10:55:55 -05:00
pub mod obj;
tamer: Introduce span This is an initial implementation optimized for expected use cases. Hopefully that pans out and doesn't come back to bite me. Regarding the context: it only allows for interned paths atm, which are strings (and so much be valid UTF-8, which is fine for us, but sucks for something more general-purpose). I'll be curious if the context needs extension later on, or if different contexts will be stored in IRs (e.g. to store a template application site as well as the location of the expansion within the template body).
2021-08-13 14:59:25 -04:00
pub mod span;
tamer: Global interners This is a major change, and I apologize for it all being in one commit. I had wanted to break it up, but doing so would have required a significant amount of temporary work that was not worth doing while I'm the only one working on this project at the moment. This accomplishes a number of important things, now that I'm preparing to write the first compiler frontend for TAMER: 1. `Symbol` has been removed; `SymbolId` is used in its place. 2. Consequently, symbols use 16 or 32 bits, rather than a 64-bit pointer. 3. Using symbols no longer requires dereferencing. 4. **Lifetimes no longer pollute the entire system! (`'i`)** 5. Two global interners are offered to produce `SymbolStr` with `'static` lifetimes, simplfiying lifetime management and borrowing where strings are still needed. 6. A nice API is provided for interning and lookups (e.g. "foo".intern()) which makes this look like a core feature of Rust. Unfortunately, making this change required modifications to...virtually everything. And that serves to emphasize why this change was needed: _everything_ used symbols, and so there's no use in not providing globals. I implemented this in a way that still provides for loose coupling through Rust's trait system. Indeed, Rustc offers a global interner, and I decided not to go that route initially because it wasn't clear to me that such a thing was desirable. It didn't become apparent to me, in fact, until the recent commit where I introduced `SymbolIndexSize` and saw how many things had to be touched; the linker evolved so rapidly as I was trying to learn Rust that I lost track of how bad it got. Further, this shows how the design of the internment system was a bit naive---I assumed certain requirements that never panned out. In particular, everything using symbols stored `&'i Symbol<'i>`---that is, a reference (usize) to an object containing an index (32-bit) and a string slice (128-bit). So it was a reference to a pretty large value, which was allocated in the arena alongside the interned string itself. But, that was assuming that something would need both the symbol index _and_ a readily available string. That's not the case. In fact, it's pretty clear that interning happens at the beginning of execution, that `SymbolId` is all that's needed during processing (unless an error occurs; more on that below); and it's not until _the very end_ that we need to retrieve interned strings from the pool to write either to a file or to display to the user. It was horribly wasteful! So `SymbolId` solves the lifetime issue in itself for most systems, but it still requires that an interner be available for anything that needs to create or resolve symbols, which, as it turns out, is still a lot of things. Therefore, I decided to implement them as thread-local static variables, which is very similar to what Rustc does itself (Rustc's are scoped). TAMER does not use threads, so the resulting `'static` lifetime should be just fine for now. Eventually I'd like to implement `!Send` and `!Sync`, though, to prevent references from escaping the thread (as noted in the patch); I can't do that yet, since the feature has not yet been stabalized. In the end, this leaves us with a system that's much easier to use and maintain; hopefully easier for newcomers to get into without having to deal with so many complex lifetimes; and a nice API that makes it a pleasure to work with symbols. Admittedly, the `SymbolIndexSize` adds some complexity, and we'll see if I end up regretting that down the line, but it exists for an important reason: the `Span` and other structures that'll be introduced need to pack a lot of data into 64 bits so they can be freely copied around to keep lifetimes simple without wreaking havoc in other ways, but a 32-bit symbol size needed by the linker is too large for that. (Actually, the linker doesn't yet need 32 bits for our systems, but it's going to in the somewhat near future unless we optimize away a bunch of symbols...but I'd really rather not have the linker hit a limit that requires a lot of code changes to resolve). Rustc uses interned spans when they exceed 8 bytes, but I'd prefer to avoid that for now. Most systems can just use on of the `PkgSymbolId` or `ProgSymbolId` type aliases and not have to worry about it. Systems that are actually shared between the compiler and the linker do, though, but it's not like we don't already have a bunch of trait bounds. Of course, as we implement link-time optimizations (LTO) in the future, it's possible most things will need the size and I'll grow frustrated with that and possibly revisit this. We shall see. Anyway, this was exhausting...and...onward to the first frontend!
2021-08-02 23:54:37 -04:00
pub mod sym;
tamer: Introduce tpwrap module to contain quick_xml::Error adapter This adapter exists to implement PartialEq so that it can be derived on Error objects. This is used primarily (well, exclusively atm) for tests.
2021-07-23 23:21:19 -04:00
pub mod tpwrap;
TAMER: Extract quick_xml event-related mocks
2020-02-13 11:50:18 -05:00
#[cfg(test)]
pub mod test;