This is the culmination of a great deal of work over the past few
weeks. Indeed, this change has been prototyped a number of different ways
and has lived in a stash of mine, in one form or another, for a few weeks.
This is not done just yet---I have to finish moving the index out of Asg,
and then clean up a little bit more---but this is a significant
simplification of the system. It was very difficult to reason about prior
approaches, and this finally moves toward doing something that I wasn't sure
if I'd be able to do successfully: formalize scope using AirAggregate's
stack and encapsulate indexing as something that is _supplemental_ to the
graph, rather than an integral component of it.
This _does not yet_ index the AirIdent operation on the package itself
because the active state is not part of the stack; that is one of the
remaining changes I still have stashed. It will be needed shortly for
package imports.
This rationale will have to appear in docs, which I intend to write soon,
but: this means that `Asg` contains _resolved_ data and itself has no
concept of scope. The state of the ASG immediately after parsing _can_ be
used to derive what the scope _must_ be (and indeed that's what
`asg::air::test::scope::derive_scopes_from_asg` does), but once we start
performing optimizations, that will no longer be true in all cases.
This means that lexical scope is a property of parsing, which, well, seems
kind of obvious from its name. But the awkwardness was that, if we consider
scope to be purely a parse-time thing---used only to construct the
relationships on the graph and then be discarded---then how do we query for
information on the graph? We'd have to walk the graph in search of an
identifier, which is slow.
But when do we need to do such a thing? For tests, it doesn't matter if
it's a little bit slow, and the graphs aren't all that large. And for
operations like template expansion and optimizations, if they need access to
a particular index, then we'll be sure to generate or provide the
appropriate one. If we need a central database of identifiers for tooling
in the future, we'll create one then. No general-purpose identifier lookup
_is_ actually needed.
And with that, `Asg::lookup_or_missing` is removed. It has been around
since the beginning of the ASG, when the linker was just a prototype, so
it's the end of TAMER's early era as I was trying to discover exactly what I
wanted the ASG to represent.
DEV-13162
This is in the same spirit as previous commits modifying (or removing)
tests and benchmarks related to accessing the ASG and its indexes directly.
With this change, only `asg::air` uses the indexing and lookup methods on
`Asg`. This will allow me to extract the index from `Asg` entirely and have
`Air` solely responsible for lookup; the graph will be responsible only for,
well, being a graph. Indexing is an optimization strategy.
More information in the commit to follow. But notice how this moving
environment-related concerns away from `Asg` and into AIR, and how the
remaining environment concerns are index-related.
But there is one remaining barrier: to fully move the indexing away from
`Asg`, we have to use an alternative (and complete)
abstraction---AirAggregateCtx with its ability to resolve and introduce
scope based on the stack. The `AirIdent` token subset doesn't yet do that,
and all the work up to this point was in prepartion for doing that. Since
introducing indexing at Root a few commits ago, it's now possible to
proceed.
DEV-13162
This, finally, introduces identifier pooling in the global environment,
represented by `Root`. All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.
As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.
It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots. I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.
More to come. I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports. `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root
them.
DEV-13162
Okay, this is finally distilling into something fairly simple and
reasonable, but I'm not quite there yet.
In particular, the responsibility is simply between `Asg` (as the owner of
the index) and `AirAggregateCtx` (as the owner of the stack frames from
which environments and scope are derived). This was inevitable and I was
waiting for it, but now I have a good idea of how to clean it up and
proceed.
This also doesn't index in root yet (`active_rooting_oi` is still `None` for
`Root`), and I think I may remove `Pool` and just make it `Visible` at that
point, since it won't be going any further anyway. I don't think the
distinction is meaningful and will just complicate implementations.
The tests also need some more cleanup---the assertions ideally would live in
independent tests, and the assertion failure is in a function call rather
than the test (function) itself, so requires a Rust backtrace to locate the
line number of (unless you look at the failure data).
So I suppose this is more of a mental synchronization point than
anything. Nothing's broken, though.
DEV-13162
There's a lot of documentation on this in the commit itself, but this stems
from
a) frustration with trying to understand how the system needs to operate
with all of the objects involved; and
b) recognizing that if I'm having difficulty, then others reading the
system later on (including myself) and possibly looking to improve upon
it are going to have a whole lot of trouble.
Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months. This finally begins to formalize
that, out of frustration with package imports. But it will be a weight
lifted off of me as well, with issues of scope always looming.
This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped. Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior. There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.
See the documentation introduced by this commit for more information on
that part of this commit.
Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API. This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with. The
documentation includes clarifying remarks that hopefully make it all
understandable.
DEV-13162
This change requires every package to have a canonical name, and performs
namespec canonicalization on imports.
Since all package names are canonicalized, this opens the door to being able
to index package names at import, allowing the object to be shared on the
graph and properly reference a package after it has been resolved.
Note that the system tests' canonicalization is relative to the hard-coded
`/TODO` presently; that will change in the near future once `tamec`
generates names from the provided path.
DEV-13162
This introduces, but does not yet integrate, `CanonicalName`, which not only
represents canonicalized package names, but handles namespec resolution.
The term "namespec" is motivated by Git's use of *spec (e.g. refspec)
referring to various ways of specifying a particular object. Names look
like paths, and are derived from them, but they _are not paths_. Their
resolution is a purely lexical operation, and they include a number of
restrictions to simplify their clarity and handling. I expect them to
evolve more in the future, and I've had ideas to do so for quite some time.
In particular, resolving packages in this way and then loading the from the
filesystem relative to the project root will ensure that
traversing (conceptually) to a parent directory will not operate
unintuitively with symlinks. The path will always resolve unambigiously.
(With that said, if the symlink is to a shared directory with different
directory structures, that doesn't solve the compilation problem---we'll
have to move object files into a project-specific build directory to handle
that.)
Span Slicing
------------
Okay, it's worth commenting on the horridity of the path name slicing that
goes on here. Care has been taken to ensure that spans will be able to be
properly sliced in all relevant contexts, and there are plenty of words
devoted to that in the documentation committed here.
But there is a more fundamental problem here that I regret not having solved
earlier, because I don't have the time for it right now: while we do have
SPair, it makes no guarantees that the span associated with the corresponding
SymbolId is actually the span that matches the original source lexeme. In
fact, it's often not.
This is a problem when we want to slice up a symbol in an SPair and produce
a sensible span. If it _is_ a source lexeme with its original span, that's
no problem. But if it's _not_, then the two are not in sync, and slicing up
the span won't produce something that actually makes sense to the user. Or,
worse (or maybe it's not worse?), it may cause a panic if the slicing is out
of bounds.
The solution in the future might be to store explicitly the state of an
SPair, or call it Lexeme, or something, so that we know the conditions under
which slicing is safe. If I ever have time for that in this project.
But the result of the lack of a proper abstraction really shows here: this
is some of the most confusing code in TAMER, and it's really not doing
anything all that complicated. It is disproportionately confusing.
DEV-13162
NOTE: This temporarily breaks `tameld`. It is fixed in a future commit when
names are bound. This was an oversight when breaking apart changes into
separate commits, because the linker does not yet have system tests like
tamec does.
This is preparing for a full transition to requiring a canonical package
name. The previous `Unnamed` variant has been removed and `AirAggregate`
will provide a default `WS_EMPTY` name, as `Pkg` had done before.
The intent of this change is to allow for consulting the index before a
new `Pkg` object is created on the graph, but we're not quite ready for that
yet.
Well, that's not entirely true---the linker can be ready for that. But the
compiler needs to canonicalize import paths relative to the active package
canonical name, which it can't even do yet because tamec isn't generating a
name.
So maybe the linker will be first; it's useful to have that in a separate
commit anyway to emphasize the change.
DEV-13162
The previous commit introduced canonical names, and this uses them to index.
The next step will be to utilize those names to look up packages on
definition rather than creating a new package node, so that references to
yet-to-be-defined (or yet-to-be-imported) packages can be resolved on the
graph.
DEV-13162
This is already a concept in the XSLT-based compiler, where each package has
a `package/@name` generated from its path. The same will happen with tamec.
Before we can load packages into the graph, we need canonical identifiers so
that they can be indexed. The next commit will handle indexing using this
information.
DEV-13162
This commit includes plenty of documentation, so you should look there.
It's desirable to describe the sorting that TAME performs as a topological
sort, since that's the end result we want. This uses the ontology to
determine what to do to the graph when a cycle is encountered. So
technically we're sorting a graph with cycles, but you can equivalently view
this as first transforming the graph to cut all cycles and then sorting it.
For the sake of trivia, the term "cut" is used for two reasons: (1) it's an
intuitive visualization, and (2) the term "cut" has precedence in logic
programming (e.g. Prolog), where it (`!`) is used to prevent
backtracking. We're also preventing backtracking, via a back edge, which
would produce a cycle.
DEV-13162
tameld isn't yet adding edges to Idents from their associated Pkg (see
previous commit), but this formalizes how the ontology will interpret such a
relationship. The idea is that Idents are always owned by Pkgs, but they
may be optionally explicitly rooted, which will be used by a particular type
of DFS walk that is about to be written, which can ignore Root->Pkg and
focus instead on cross edges to Idents.
Though it's not lost on me that now that I'll be introducing a DFS for the
linker, the terms "cross" and "tree" edge now become ambiguous; I used to
call them "ontological X edge", but I had fallen out of that habit; perhaps
I need to reintroduce that rigor.
DEV-13162
This may now index _any_ type of object, in preparation for indexing package
import paths. In practice, this only makes sense (at least currently) for
`Pkg` and `Ident`.
This generalization also applies to `Asg::lookup_or_missing`.
DEV-13162
Historically, the ASG was better described as a "dependency graph",
containing only identifiers (which are simply called "symbols" in the
XSLT-based compiler). Consequently, it was appropriate for the graph to
have operations specific to identifiers. (Indeed, that's the only type of
object the graph supported.)
Much has changed since then. This cleans things up, and makes parenting
identifiers to root an _explicit_ operation. This will make it easier to
move forward with handling of scope, and importing identifiers into
packages, and removing `Source`, and so on.
DEV-13162
I've been torturing myself trying to figure out how I want to generalize
indexing, lookups, and value numbering in a way that is appropriate for this
project (that is, not over-engineered relative to my needs).
Before I can do much of anything, though, I need to stop having indexing
only as a `Root` thing (previously it wasn't even tied to `Root`). This
makes that change for tamec, but temporarily removes scoping concerns until
I can add more specific types of indexing.
Not only does this allow cleaning up some `Ident`-specific stuff from `Asg`,
but the cleanup also helps to show that portions of the system aren't still
using Root-based globals.
The linker (`tameld`) still uses the old `global` methods for now; those
will eventually go away, but this needs to change to unify both tamec and
tameld once we get to imports as part of the compiler.
DEV-13162
This is needed to then support `@desc` for shorthand desugaring; it's
required by the XSLT-based compiler (and will eventually be required by
TAMER too).
DEV-13708
This supports arbitrary documentation as sibling text (mixed content, in XML
terms). The motivation behind this change is to permit existing system
tests to succeed when `Todo | TodoAttr` are both rejected, rather than
having to ignore this.
TAME has always had a philosophy of literate documentation, however it was
never fully realized. This just maintains the status quo; the text is
unstructured, and maybe will be parsed in the future.
Unfortunately, this does _not_ include the output in the `xmli` file or the
system tests. The reason has nothing to do with TAMER---`xmllint` does not
format the output when there is mixed content, it seems, and I need to move
on for now; I'll consider my options in the future. But, it's available on
the graph and ready to go.
DEV-13708
This introduces a new `Doc` object that can be owned by `Expr` (only atm)
and contain what it describes as a concise independent clause. This
construction is not enforced, and is only really obvious today via the
Summary Pages.
There's a lot of latent and unrealized potential in TAME's documentation
philosophy that was never realized, so this will certainly evolve over
time. But for now, the primary purpose was to get `@desc` working on things
like classifications so that `xmli` output can compile for certain
packages.
DEV-13708
These are used by virtually every `ObjectKind`; I've been meaning to do this
for a while, but now that I'm about to introduce a new one (`Doc`), let's
just get it out of the way.
DEV-13708
This doesn't do the actual hard work yet of resolving and loading a package,
but it does place it on the graph and re-derive it into the xmli output.
DEV-13708
This introduces `<match on="foo" />` and `<match on="foo" value="bar" />`,
which are both equality predicates. Other types of predicates are not yet
supported.
This change is a bit messy and leaves a bit to be desired. `NirToAir` is
quite messy and needs some cleanup. There's also the issue of introducing
XML-specific errors in NIR so that users know what things like "subject"
mean, but not being able to do so yet because NIR is agnostic to the source
document type; another layer of abstraction is needed.
But, my priority is first to get derivation of a particularly
expensive (generated) package in our internal systems working first.
DEV-13708
This recognizes template application within expressions. Since expressions
can occur within templates, this can occur arbitrarily deeply.
And with that, we have the core of the template system represented on the
graph. Of course, there are some glaring scoping issues to be resolved, but
those aren't unique to template application.
DEV-13708
I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
All ObjectIndex-like objects hash using only the underlying identifier,
which ultimately boils down to a `NodeIndex` (petgraph), which is just a
u32. And so in that sense, the only purpose we have for hashing it is to
(a) reduce the space required to store mappings, and (b) compose with other
`Hash`es.
DEV-13708
This creates another trait and struct `ObjectIndexToTree` that assert a
stronger invariant than `ObjectIndexRelTo`---that not only does it uphold
the invariants of `ObjectIndexRelTo`, but also that it represents a _tree_
edge, which indicates _ownership_ rather than just a reference.
This will be used to statically infer what can serve as a scope boundary for
upcoming changes. Specifically, anything that can own an `Ident` introduces
a new level of scope.
DEV-13708
This allows this method to be used on anything that is able to relate to an
identifier, which is needed for the changes being made for the template
system.
This linear lookup is actually going away (as hinted at by preceding
commits); this is extracted as part of a larger change and I wanted to get
it committed to make it easier to follow upcoming changes.
DEV-13708
This is more of the same of the previous commit, but in a more digestable
chunk. We now have child states that are able to be constructed using a
simple `From`, which is important to making `AirAggregate` a `SuperState`.
This also makes `AirStack` act like a prototype chain for `ObjectIndex`es,
creating environments where context shadows. The linear search should only
have to check the last two frames (e.g. an Expr has a parent Pkg or Tpl
context which will have a `rooting_oi` value), and this is only done during
a rooting operation.
DEV-13708
This begins to introduce `AirStack` and starts to migrate context away from
the individual `ParseState`s onto the stack.
I should have started to commit earlier; this is getting a bit large and
makes it hard to follow what I'm doing so, hopefully stopping a little bit
short will allow the following commit to show that.
This is a work-in-progress change. All tests pass, but the refactoring is
incomplete. The `AirStack` abstraction is _also_ incomplete and will have
better, more domain-specific operations that make it harder to mess up
pairing pushes with pops.
The purpose of doing this is to allow `AirAggregate` to serve exclusively as
a sum state, which can then become a SuperState, much like `ele_parse!`'s
approach.
The _end_ goal of all of this is arbitrary template nesting.
DEV-13708
The graph's ontology is defined in the direction of the edge: from OA
to OB. This is enforced by the type system to ensure that no code path is
able to generate an invalid graph.
But that also makes it very difficult to work with a generic source to a
specific target.
This introduces a `ObjectIndexRelTo` trait that says whether `Self` is able
to be related to some `ObjectKind` `OB`, implements it for `ObjectIndex
where ObjectRelTo<OB>`, and introduces a new semi-opaque type
`ObjectIndexTo` that allows for the source `ObjectIndex` to be generic.
This then redefines some existing graph primitives in terms of
`ObjectIndexRelTo`, in particular creating edges, so that `ObjectIndex` can
be used as today, and the new `ObjectIndexTo` can be used in the same way
with the same API, without violating the graph ontology.
This will be used by `AirAggregate` to create dynamic targets for rooting
and splicing/expansion.
DEV-13708
Previously, `AirTplAggregate` worked only in a `Pkg` context, being able to
root `Tpl` `Ident`s in `Pkg` and expand only into `Pkg`. This still does
the same, but generalizes to allow for different roots and expansion
targets.
This will be utilized to parse nested templates.
DEV-13708
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least). And, as you can see, there's still a lot of TODO items here.
I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.
But, it works.
DEV-13708
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.
Templates must not contribute to this environment until expansion. Further,
metavariables will not be present in this environment. To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed. This will also allow for the creation of more general
methods down the line.
DEV-13708
Previous to this commit, ontological cross edges were declared
statically. But this doesn't fare well with the decided implementation for
template application.
The documentation details it, but we have Tpl->Ident which could mean "I
define this Ident once expanded", or it could mean "this is a reference to a
template I will be applying". The former is a tree edge, the latter is a
cross edge, and that determination can only be made by inspecting edge data
at runtime.
It could have been resolved by introducing new Object types, but that is a
lot of work for little benefit, especially given that only (right now) the
visitor uses this information.
DEV-13708
This this a big change that's difficult to break up, and I don't have the
energy after it.
This introduces nullary template application, short- and long-form. Note
that a body of the short form is a `@values@` argument, so that's not
supported yet.
This continues to formalize the idea of what "template application" and
"template expansion" mean in TAMER. It makes a separate `TplApply`
unnecessary, because now application is simply a reference to a
template. Expansion and application are one and the same: when a template
expands, it'll re-bind metavariables to the parent context. So in a
template context, this amounts to application.
But applying a closed template will have nothing to bind, and so is
equivalent to expansion. And since `Meta` objects are not valid outside of
a `Tpl` context, applying a non-closed template outside of another template
will be invalid.
So we get all of this with a single primitive (getting the "value" of a
template).
The expansion is conceptually like `,@` in Lisp, where we're splicing trees.
It's a mess in some spots, but I want to get this committed before I do a
little bit of cleanup.
Also known as metavariables or template parameters.
This is a bit of a tortured excursion, trying to figure out how I want to
best represent this. I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.
`AirTpl::TplApply` highlights some remaining questions. What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such. But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue. But I'd rather address that head-on in
the future.
So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time. This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.
My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them. That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.
But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems
viable.
DEV-13708
Small break from templates for something easier. I have COVID-19, so I'll
use that as my excuse for wanting to be more lazy.
The real reason is to see some more concrete progress and ensure that
patterns hold for simple expressions before further refactoring.
But, before I proceed with such refactoring, I really ought to approach
something that requires a NIR desugaring step, like case statements.
DEV-13708
This sets us up to be able to determine how `Dangling` expressions will be
rooted into templates.
This new strategy isn't yet handling `Dangling`; I wanted to get this
committed first so that the `Dangling` refactoring is more clear.
DEV-13708
This sets the stage for template parsing, and finally decides how we're
going to represent templates on the ASG. This is going to start simple,
since my original plans for improving how templates are
handled (conceptually) is going to have to wait.
This is the last difficult object type to figure out, with respect to graph
representation and derivation, so I wanted to get it out of the way.
DEV-13708
The previous commit demonstrated the amount of boilerplate necessary for
introducing new `ObjectKind`s; this abstracts away a lot of that
boilerplate, and allows for declarative relationship definition for the
ASG's ontology.
DEV-13708
There's quite a bit of boilerplate here that'll eventually need factoring
out. But it's also clear that it is somewhat onerous to add new object
types.
Note that a good chunk of this burden is _intentional_, via exhaustiveness
checks---adding a new type of object is an exceptional occurrence (well, in
principle, but we haven't added them all yet, so it'll be more common
initially), and we'd rather be safe to ensure that everything is properly
considering how that new type of object interacts with it.
Let's not confuse coupling with safety---the latter causes a burden because
of the former, not because of itself; it provides a service to us.
But, nonetheless, we'll want to reduce this burden somewhat since there are
a number more to add.
DEV-13708
Just as `rate` is a `sum`, `classify` is an `all` by default. The `@any`
attribute will change that interpretation, though I only intend to recognize
that in parsing later on, not emit that in XMLI.
DEV-13708
This is generic over the source, just as the target, defaulting just the
same to `ObjectIndex`.
This allows us to use only the edge information provided rather than having
to perform another lookup on the graph and then assert that we found the
correct edge. In this case, we're dealing with an `Ident->Expr` edge, of
which there is only one, but in other cases, there may be many such edges,
and it wouldn't be possible to know _which_ was referred to without also
keeping context of the previous edge in the walk.
So, in addition to avoiding more indirection and being more immune to logic
bugs, this also allows us to avoid states in `AsgTreeToXirf` for the purpose
of tracking previous edges in the current path. And it means that the tree
walk can seed further traversals in conjunction with it, if that is so
needed for deriving sources.
More cleanup will be needed, but this does well to set us up for moving
forward; I was too uncomfortable with having to do the separate
lookup. This is also a more intuitive API.
But it does have the awkward effect that now I don't need the pair---I just
need the `Object`---but I'm not going to remove it because I suspect I may
need it in the future. We'll see.
The TODO references the fact that I'm using a convenient `resolve_oi_pairs`
instead of resolving only the target first and then the source only in the
code path that needs it. I'll want to verify that Rust will properly
optimize to avoid the source resolution in branches that do not need it.
DEV-13708
This makes the inner `Object` type generic (but defaulting to the same inner
types as before) so that it can be used as a sum type for various types
where `ObjectKind`-based narrowing is required.
In this case, it's used to narrow `ObjectIndex` alongside the inner
`ObjectKind` so that the two are definitely in sync. This not only results
in cleaner code and a more intuitive API that's approachable to people
less familiar with the system, but it also helps to eliminate logic bugs
that might result form manually narrowing (as was done before this change).
DEV-13708
This extends the POC a bit by beginning to reconstruct rate blocks (note
that NIR isn't producing sub-expressions yet).
Importantly, this also adds the first system tests, now that we have an
end-to-end system. This not only gives me confidence that the system is
producing the expected output, but serves as a compromise: writing unit or
integration tests for this program derivation would be a great deal of work,
and wouldn't even catch the bugs I'm worried most about; the lowering
operation can be written in such a way as to give me high confidence in its
correctness without those more granular tests, or in conjunction with unit
or integration tests for a smaller portion.
DEV-13708
Just some final POC setup for how this'll work; it's nothing
significant. This just emits an `@xmlns` on the `package` element to
demonstrate use of the stack.
With that, it's time to formalize this.
I also need to document at some point why I choose to use `ArrayVec` still
over `Vec`---it's not a microoptimization. It's intended to simplify the
runtime to keep execution simple with fewer code paths and make it more
amenable to analysis. Memory allocation is a pretty complex thing and
muddies execution. It's also another point of failure, though practically
speaking, I'm not worried about that---this is replacing a system that
consumes many GiB of memory (XSLT-based compiler) with one that consumes 10s
of MiB.
DEV-13708
This (a) hold the state of a stack that I can populate with tokens rather
than introducing a state for every single e.g. attribute and such on
elements (so, more like the `xmle` XIR lowering).
It also hides the obvious awkwardness of the `&mut &'a Asg`, but that's not
the intent of it.
DEV-13708
This has been a long time coming. The wiring of it all together is a little
rough around the edges right now, but this commit represents a working POC
to begin to fill in the gaps for the entire lowering pipeline.
I had hoped to be at this point a year ago. Yeah.
This marks a significant milestone in the project because this allows me to
begin to observe the implementation end-to-end, testing it on real-life
inputs as part of a production build pipeline.
...and now, with that, we can begin. So much work has gone into this
project so far, but aside from the linker (which has been in production for
years), most of this work has been foundational. It's been a significant
investment that I intend to have pay off in many different ways.
(All this outputs right now is `<package/>`.)
DEV-13708
Mike Gerwitz