Compare commits


146 Commits

Author SHA1 Message Date
Mike Gerwitz 6769f0c280 tamer: Support nightly Rust toolchain pinning
I had never intended to avoid pinning nightly.  This is an unfortunate thing
to have to do---require a _specific_ version of a compiler to build your
software; it's madness.  But the unstable features utilized by TAMER (as
rationalized in `src/`) are still worth the effort.

It's not _actually_ that case that we need a specific version of the
compiler, granted; this is outlined in `rust-toolchain.toml`'s
rationale.  You should look there for more information; my approach still
utilizes explicit channels via cargo.  Unfortunately, I had hard-coded it
previously, putting me in a bit of a bind an unable to override the behavior
without modifying the software.

The reason for this change is that `adt_const_params` has a BC break
involving the introduction of `ConstParamTy`.  This is only the second time
I've been bitten by a nightly BC break; the other was the renaming of
`int_log`'s API, as mentioned in
709291b107.  This pinning will in fact
mitigate those future issues---TAMER will be able to resolve the issue at
its leisure, and will further be able to continue to build earlier commits
in the future by simply re-bootstrapping with the committed nightly

If you're curious of my rationale for wanting to inhibit toolchain
downloading during build, or use system libraries, have a look at GNU Guix's
approach to building software safely and reproducibly.  In particular,
dependencies are also built from source (rather than downloading binaries
from external sources), and builds take place in network-isolated
containers.  The `TAMER_RUST_TOOLCHAIN` configure parameter is meant to
facilitate these situations by giving more flexibility to packagers.

2023-06-05 16:42:31 -04:00
Mike Gerwitz 1706c55645 tamer: nir::air: Feature-flag SYM_TRUE
The code utilizing this is flagged, and so the build would output warnings
saying that it was not used.  This resolves that (I've been aware of it for
far too long; I'm developing behind the `wip-asg-derived-xmli` flag where I
don't usually see it).

2023-06-05 16:27:56 -04:00
Mike Gerwitz 93cc0d2ce1 tamer: pipeline::macro::lower_pipeline: Doc generation
This generates some documentation helping to describe the lowering pipeline,
since the function type signature can be daunting to those unfamiliar with
it (and I'm sure to the future me too).

2023-06-05 13:44:49 -04:00
Mike Gerwitz 65c1b2d083 tamer: pipeline: Remove explicit source token type specification
Like the previous commit's removal of the error type, this eliminates the
explicit source token type since we're able to infer it from the pipeline

2023-06-05 13:44:49 -04:00
Mike Gerwitz 0e0f3e658d tamer: pipeline: Remove explicit error specification in pipeline definition
It does not matter what the error of the source is as long as the caller is
able to deal with it, especially given that the particular error is a
property of the source, which is under control of the caller.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 3800109530 tamer: pipeline: Extract macro into own module
The macro is off-putting and more complicated than the pipeline definitions
themselves (of course), so this tucks it away so that readers are able to
more easily observe the definitions that they're probably looking for
without feeling compelled to try to understand the macro definition.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 6a99ee3cb3 tamer: pipeline::lower_xmli: Use `lower_pipeline!`
All lowering pipelines are now using `lower_pipeline!`.  Finally.

The macro does require some refactoring and documentation, but it's working,
and we now have three pipelines whose definitions are smaller than a single
one was previously.  I've been hoping to do this for many months, so it's
nice to finally see this come to fruition.

I had been putting it off, but doing so has made it difficult to compose
other parts of the system, not knowing what abstractions I'll have at my

2023-06-05 13:44:49 -04:00
Mike Gerwitz 109ba5f797 tamer: pipeline::lower_xmli: Generalize sink like other pipelines
This makes the sink similar to other pipelines without creating a new
ParseState, and so will allow for integrating into the `lower_pipeline!`

2023-06-05 13:44:49 -04:00
Mike Gerwitz 9c6b00a124 tamer: pipeline: Initial concept for declarative pipeline definition
This has been the ultimate goal for the pipeline for some time---the ability
to declaratively define the lowering pipeline in a way that is clear,
concise, and is correct by definition.

The reason that the lowering pipeline required so much boilerplate was
because of the robust types involved, which ensures that everything in the
pipeline is compatible with one-another---it's not possible to construct a
pipeline that will not work.

Of course, there is nuance involved in some cases---I didn't want to include
the `until` clause, which makes it fail the "obviously correct" criterion,
but that can be improved over time.

This only abstracts away `load_xmlo` and `parse_package_xml`; next I'll have
to evolve the abstraction to support lifetimes for `lower_xmli`'s
`AsgTreeToXirf`.  That pipeline also ends with a custom sink that really
ought to become its own parser, but I don't want to jump down that rabbit
hole right now, so we may just support custom sinks for now with the intent
of removing it in the future.

This has been a long time coming.  The ultimate goal is that you should be
able to look at the parser pipelines to have a clear, high-level overview of
how everything fits together.  I'm not generating documentation yet, but
that'll help serve as a guide as well.

2023-06-05 13:44:49 -04:00
Mike Gerwitz f34f2644e9 tamer: pipeline: Allow reporting on entire Result
The report acts as the sink for `load_xmlo` and `parse_package_xml`.  At the
moment, the type is `()`, and so there's nothing to report on but the
error.  But the idea is to add logging via `AirAggregate::Object`, which is
currently just `()`.

This change therefore is only a refactoring---it changes no functionality
but sets up for future changes.

This also introduces consistency with `lower_xmli` in use of `terminal` for
the final operation.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 0f20f22cfb tamer: diagnose::Diagnostic: Remove Error trait bound
Diagnostic events need not be errors.  While that was the original intent,
it'd also be nice to be able to use the diagnostic system for any type of
logging, where the verbosity level would determine the type of report that
is output (whether source information should be provided).

Then we could have e.g. AirAggregate produce events describing what actions
are occurring, which could be much more useful than a trace in many
contexts, and would be able to operate via a runtime toggle/filter without
having an adverse effect on performance (since the diagnostic rendering
itself is the hit; the underlying data are cheap).

Anyway---I'm addressing this now to generalize the reporter in the lowering
pipeline, so that it can report on not just errors but anything.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 2bf3122402 tamer: pipeline::load_xmlo: Hoist context decomposition and format
This formats the pipeline to mirror the style of
`parse_package_xml`.  Based on the previous commits, the end goal (though
not necessarily now) will be to derive a concise abstraction for all the
lowering pipelines, which means first factoring them into a common form.

2023-06-05 13:44:49 -04:00
Mike Gerwitz b5187de5dc tamer: pipeline::load_xmlo: Accept reporter
This makes the API of `load_xmlo` much closer to `parse_package_xml`, both
accepting a reporter and distinguishing between recoverable and
unrecoverable errors.

The linker still does not use a reporter and still fails on the first
error, as before; I wanted to keep this change small.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 896fb3a0e5 tamer: asg::air::ir::AirPkg::PkgImport: New token
This allows us to drop `AirIdent::IdentRef`, which in turn allows dropping
`AirIdent` entirely from `AirPkgAggregate`.

This is also a more appropriate abstraction; having to track all the ways in
which `IdentRef` was used can be confusing.  This means that `AirIdent` is
true to its name---used only for identifiers.  The new token type makes it
very clear where package imports are recognized, and it's also easier to
search for.

2023-06-05 13:44:49 -04:00
Mike Gerwitz 1f2315436c tamer: tamec: Extract xmli lowering into pipeline module
This is the same idea as the previous two commits: get all the lowering
pipelines into the same place so that we can observe commonalities and
attempt to derive an appropriate abstraction.

`lower_xmli` could have invoked `tree_reconstruction` itself, since it has
all the information that it needs to do so, but the idea is that these will
accept sources from the caller.  This also demonstrates that sinks need to
be flexible.  In an ideal abstraction, perhaps this would be able to produce
an iterator that accepts the first token type and yields the last, which can
then be directed to a sink, but that's not compatible with how the lowering
operations currently work, which requires a single value to be
returned.  But if it did work that way, then they'd be able to compose just
as any other parser.

Maybe for the future.

2023-06-05 13:44:48 -04:00
Mike Gerwitz 9e8b809c14 tamer: tamec: Extract package parsing into pipeline module
The previous commit extracted xmlo loading, because that will be a common
operation between `tamec` and `tameld`.  This extracts parsing, which will
only used by `tamec` for now, though components of the pipeline are similar
to xmlo loading.

Not only does it need to be removed from `tamec` and better abstracted, but
the intent now is to get all of these things into one place so that the
patterns are obviated and a better abstraction can be created to remove all
of this boilerplate and type complexity.

Furthermore, xmlo loading needs to use reporting and recovery, so having
`parse_package_xml` here will help show how to make that happen easily.  I'm
pleased that it ended up being trivial to extract error reporting from the
lowering pipeline as a simple (mutable) callback.  I'm not pleased about
the side-effects, but, this works well for now given how the system works

2023-06-05 13:44:46 -04:00
Mike Gerwitz 57a805b495 tamer: src::pipeline: Eliminate most error type references
Just cleaning up a bit, removing some unnecessary types, since there are so
many involved.

2023-05-25 16:58:44 -04:00
Mike Gerwitz ea6259570e tamer: ld::poc: Extract xmlo loading pipeline into new pipeline module
I want to clean this up a bit further.  The motivation is that we need this
for imports in `tamec`.

Eventually this will be cleaned up to the point where it's declarative and
easy to understand---there's a mess of types involved now and, when
something goes wrong, it can be brutally confusing.

2023-05-25 16:38:41 -04:00
Mike Gerwitz 4ac8bf5981 tamer: asg::air: Isolate scope boundary rules
This extracts and decouples the boundary rules from the stack frames
themselves, which not only clarifies what the rules are (and makes them
match the scope diagrams), but paves the way for future isolation.

2023-05-24 15:31:10 -04:00
Mike Gerwitz 294caaa35a tamer: asg::graph::object: Remove declare_local
This was used for metavariable declaration before scoping was sorted
out.  That was just resolved, and so this is no longer needed (and is indeed
not desirable, since it side-steps the scope index and so will not be found
except by `lookup_local_linear`).

2023-05-24 14:56:22 -04:00
Mike Gerwitz 19a5ec1e0f tamer: asg: Reduce Debug output of `Asg` and `AirAggregateCtx`
The ASG had its output reduced previously but I had apparently stashed it; I
found it while trying to clean up after so many failed or partial attempts
and the various scoping changes.

The most fundamental issue is that there's too much information: it's very
difficult to interrogate so I seldom look at it, and it slows down Parser
trace output to the point where it's useless on even one of our smallest
systems, generating 1.5GiB of output for a graph of ~10k
objects (via tameld).

2023-05-23 16:15:38 -04:00
Mike Gerwitz ac9b7f620e tamer: asg::air: Remove comment about AIR being light
It used to be, but has not been for quite some time.  Further, it _does_
replace (encapsulate, rather), the ASG's API.

2023-05-23 14:46:09 -04:00
Mike Gerwitz e8335c57d4 tamer: asg::air::ir::AirMeta: Remove `Tpl` prefix from tokens
Cleanup from the previous commit.

2023-05-23 14:44:16 -04:00
Mike Gerwitz c12bf439ae tamer: asg::air: Index scope of local metavariables
The scope system works with the AIR stack frames, expecting all parent
environments to be on that stack.  Since metavariables were (awkwardly) part
of the template parser, that didn't happen.

This change extracts metavariable parsing (with some remaining TODOs) into
its own parser, so that `AirTplAggregate` will be on the stack; then it's a
simple matter of using the existing `AirAggregateCtx` methods to define a
variable and index its shadow scope, which addresses TODOs in the existing
scope test cases.

This also involved separating the tokens from `AirTpl` into `AirMeta`; they
need to be renamed, which will happen in a following commit, since this is
large enough as it is.

Another change that had to be included here, which I wish I could have just
done separately if it wasn't too much work, was to permit overlapping
identifier shadows.  Local variables have to cast a shadow so that we can
figure out if they would in turn shadow an identifier (which would be an
error), but they don't conflict with one-another if they don't have a
shared (visible) scope.

`AirAggregate` can be simplified even further, e.g. to eliminate the
expression stack and just use the ctx stack (which didn't previously exist),
but I need to continue; I'll return to it.

2023-05-23 14:38:01 -04:00
Mike Gerwitz da4d7f83ea tamer: asg::air::test::scope: Explicitly assert against indexed identifier span
That was being done automatically before this change, but the change that
I'm about to introduce for metavariables will require this distinction, at
the very least to emphasize the behavior of the indexing.

See the next commit for more information.

(The next commit has a bit too much going on, so I wanted to at least
attempt to separate things where it wasn't much work to do so.)

2023-05-23 11:59:45 -04:00
Mike Gerwitz 434365543e .gitlab-ci.yml: build: Clean before build
The motivating factor here is some out of date or corrupted rustc cache,
however we really ought to be doing fresh builds for TAME; it doesn't add
enough time that it's worth sacrificing assurances.
2023-05-19 13:43:41 -04:00
Mike Gerwitz e940fc5aa0 tamer: asg: Move index from Asg to AirAggregateCtx
This finally removes the awkward index from the ASG.  This will need much
more documentation and a better organized abstraction, but in the meantime,
previous commit dive into some of the rationale.

In essence: it only really makes sense to have indexing on the ASG itself if
it is used to cache queries or other expensive operations.  But that is not
what we were using it for---it was used for caching _lexical_ properties,
which are useful only during parsing for the sake of forming relationships
on the graph.  Once those relationships have formed, different types of
indexes will be useful in different lowering, optimization, or querying

This formalizes that, and in doing so, ensures that the index is will always
be accurate relative to the content of the ASG.  Once the index becomes
separated from it---through the `AirAggregateCtx::finish` operation---then
it is discarded and the ASG exposed.

This is also important because the index is incomplete---it contains only
the information necessary for the parser to carry out its task.

This change was a long time coming, and has reduced ASG to its essence.

2023-05-19 13:38:17 -04:00
Mike Gerwitz 7857460c1d tamer: Re-use prior AirAggreagteCtx for subsequent parsers
A new AirAggregate parser is utilized for each package import.  This
prevents us from moving the index from `Asg` onto `AirAggregateCtx` because
the index would be dropped between each import.

This allows re-using that context and solves for problems that result from
attempting to do so, as explained in the new
`resume_previous_parsing_context` test case.

But, it's now clear that there's a missing abstraction, and that reasoning
about this problem at the topmost level of the compiler/linker in terms of
internal parsing details like "context" is not appropriate.  What we're
doing is suspending parsing and resuming it later on for another package,
aggregating into the same destination (ASG + index).  An abstraction ought
to be formed in terms of that.

2023-05-19 13:38:15 -04:00
Mike Gerwitz 92214c7e05 tamer: asg::air: Include package in opaque identifier scope
This was the remaining of my stashed changes that I had mentioned in a
previous commit, but is accomplished differently than I had prototyped.  My
initial approach was a bit too klugey: to accept as an argument in various
scope contexts the active parser, as if it were the top stack frame.  This
was prototyped before the `AirPkgAggregate` parser was even created.

So we've since created a Pkg parser and now an opaque parser for opaque
idents.  There may be other opaque objects in the future.

Because of this change, the parent `AirPkgAggregate` gets stored on the
stack and just naturally becomes part of the lexical scope determination,
and so everything Just Works!

This commit was _supposed_ to be moving the index from `Asg` onto
`AirAggregateCtx`, but I wasn't able to do that because that context is
re-created for each package import currently.

2023-05-19 09:46:36 -04:00
Mike Gerwitz e266d42c48 tamer: asg::air::AirAggregateCtx: Tuple struct => named struct
As evidenced by this change, the tuple syntax was no longer serving us
well.  But the real reason for this change is to prepare for the addition of
a fourth field: the index, taken from `Asg`.

2023-05-18 01:32:28 -04:00
Mike Gerwitz 8b3dfe9149 tamer: asg::air: Utilize AirAggregateCtx for index lookups
This change means that `asg::air` is now the only module that directly
invokes index-related methods on `Asg`.  This clears the way, finally, to
removing the index from `Asg` entirely.

Not only does this result in a less awkward architecture, it also ensures
that lookups are forced to go through the system that understands and
controls lexical scoping, which will be able to give the correct answer.

Of course, the caveat is that the "correct" answer depends on what's
currently on the stack, depending on what type of lookup is being performed,
but those details are still encapsulated within the `asg::air` module and
its tests.

2023-05-18 01:10:23 -04:00
Mike Gerwitz 94bbc2d725 tamer: asg::air: Root AirIdent operations using AirAggregateCtx
This is the culmination of a great deal of work over the past few
weeks.  Indeed, this change has been prototyped a number of different ways
and has lived in a stash of mine, in one form or another, for a few weeks.

This is not done just yet---I have to finish moving the index out of Asg,
and then clean up a little bit more---but this is a significant
simplification of the system.  It was very difficult to reason about prior
approaches, and this finally moves toward doing something that I wasn't sure
if I'd be able to do successfully: formalize scope using AirAggregate's
stack and encapsulate indexing as something that is _supplemental_ to the
graph, rather than an integral component of it.

This _does not yet_ index the AirIdent operation on the package itself
because the active state is not part of the stack; that is one of the
remaining changes I still have stashed.  It will be needed shortly for
package imports.

This rationale will have to appear in docs, which I intend to write soon,
but: this means that `Asg` contains _resolved_ data and itself has no
concept of scope.  The state of the ASG immediately after parsing _can_ be
used to derive what the scope _must_ be (and indeed that's what
`asg::air::test::scope::derive_scopes_from_asg` does), but once we start
performing optimizations, that will no longer be true in all cases.

This means that lexical scope is a property of parsing, which, well, seems
kind of obvious from its name.  But the awkwardness was that, if we consider
scope to be purely a parse-time thing---used only to construct the
relationships on the graph and then be discarded---then how do we query for
information on the graph?  We'd have to walk the graph in search of an
identifier, which is slow.

But when do we need to do such a thing?  For tests, it doesn't matter if
it's a little bit slow, and the graphs aren't all that large.  And for
operations like template expansion and optimizations, if they need access to
a particular index, then we'll be sure to generate or provide the
appropriate one.  If we need a central database of identifiers for tooling
in the future, we'll create one then.  No general-purpose identifier lookup
_is_ actually needed.

And with that, `Asg::lookup_or_missing` is removed.  It has been around
since the beginning of the ASG, when the linker was just a prototype, so
it's the end of TAMER's early era as I was trying to discover exactly what I
wanted the ASG to represent.

2023-05-17 12:23:36 -04:00
Mike Gerwitz 716e217c9f tamer: asg: Restrict index-related operations to AIR
This is in the same spirit as previous commits modifying (or removing)
tests and benchmarks related to accessing the ASG and its indexes directly.

With this change, only `asg::air` uses the indexing and lookup methods on
`Asg`.  This will allow me to extract the index from `Asg` entirely and have
`Air` solely responsible for lookup; the graph will be responsible only for,
well, being a graph.  Indexing is an optimization strategy.

More information in the commit to follow.  But notice how this moving
environment-related concerns away from `Asg` and into AIR, and how the
remaining environment concerns are index-related.

But there is one remaining barrier: to fully move the indexing away from
`Asg`, we have to use an alternative (and complete)
abstraction---AirAggregateCtx with its ability to resolve and introduce
scope based on the stack.  The `AirIdent` token subset doesn't yet do that,
and all the work up to this point was in prepartion for doing that.  Since
introducing indexing at Root a few commits ago, it's now possible to

2023-05-17 11:37:03 -04:00
Mike Gerwitz 92eb991df3 tamer: benches: Remove asg and asg_lower_xmle microbenchmarks
These benchmarks were useful as TAMER was in its infancy and I was trying to
gain an intuition for working with Rust.  But they are now out of date, and
there are better ways to measure TAMER's performance, including running it
on real-world data (which wasn't possible previously) and through profiling
tools like Valgrind.

With that said, these types of benchmarks _would_ be useful for helping to
dig down into improvements that could be made, at a glance.  The problem is,
they aren't testing anything new, and they're also testing something I'm
about to extract from `Asg`.  It is not worth the ongoing maintenance cost.

So benchmarks may be reintroduced in the future if they are found to be

2023-05-17 11:14:00 -04:00
Mike Gerwitz b61e1ce952 tamer: asg::air: Common asg_from_toks for tests
The previous commit introduced a duplicate `asg_from_toks`; this just makes
it available publicly for any tests that might utilize AIR to lower the
barrier to writing such tests and provide some guidance in doing so.

2023-05-17 10:57:10 -04:00
Mike Gerwitz 79fa10f26b tamer: ld::xmle::lower::test: Use AIR (decouple from Asg and index)
This uses AIR---the ASG's proper public interface now---to construct the
graph for tests, just as all the other modern tests do.  This is change
works towards encapsulating index operations (both creation and lookups) so
that the index can be moved off of Asg and into AIR, where it belongs.  More
information on that and rationale to come.

2023-05-17 10:50:57 -04:00
Mike Gerwitz b238366bee tamer: asg::air::test::scope: Individual test per scope assertion
This separates the previous scope assertions per identifier into individual
tests, so that a single failure will not preempt all others.

2023-05-17 00:25:31 -04:00
Mike Gerwitz bfbaa6e528 tamer: asg::air::test::scope: More useful panic function/line
This will panic within the test function so that RUST_BACKTRACE is not

It also prepares for the changes coming up next.

2023-05-16 23:49:23 -04:00
Mike Gerwitz ba38a3c1ba tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`.  All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.

As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.

It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots.  I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.

More to come.  I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports.  `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root

2023-05-16 23:28:47 -04:00
Mike Gerwitz 1cf5488756 tamer: asg::air::EnvScopeKind::Pool: Remove (Visible is sufficient)
At least, I think so.

See previous commit for more information, and the commit that follows for
actually using it at Root.

2023-05-16 23:28:45 -04:00
Mike Gerwitz 33f34bf244 tamer: asg: Initial identifier scoping
Okay, this is finally distilling into something fairly simple and
reasonable, but I'm not quite there yet.

In particular, the responsibility is simply between `Asg` (as the owner of
the index) and `AirAggregateCtx` (as the owner of the stack frames from
which environments and scope are derived).  This was inevitable and I was
waiting for it, but now I have a good idea of how to clean it up and

This also doesn't index in root yet (`active_rooting_oi` is still `None` for
`Root`), and I think I may remove `Pool` and just make it `Visible` at that
point, since it won't be going any further anyway.  I don't think the
distinction is meaningful and will just complicate implementations.

The tests also need some more cleanup---the assertions ideally would live in
independent tests, and the assertion failure is in a function call rather
than the test (function) itself, so requires a Rust backtrace to locate the
line number of (unless you look at the failure data).

So I suppose this is more of a mental synchronization point than
anything.  Nothing's broken, though.

2023-05-16 14:58:21 -04:00
Mike Gerwitz 9fb2169a06 tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems

  a) frustration with trying to understand how the system needs to operate
     with all of the objects involved; and
  b) recognizing that if I'm having difficulty, then others reading the
     system later on (including myself) and possibly looking to improve upon
     it are going to have a whole lot of trouble.

Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months.  This finally begins to formalize
that, out of frustration with package imports.  But it will be a weight
lifted off of me as well, with issues of scope always looming.

This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped.  Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior.  There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.

See the documentation introduced by this commit for more information on
that part of this commit.

Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API.  This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with.  The
documentation includes clarifying remarks that hopefully make it all

2023-05-12 14:07:29 -04:00
Mike Gerwitz 00ff660008 tamer: asg::air: Begin lexical identifier resolution from bottom up
This begins demonstrating that the root will be utilized for identifier
lookup and indexing, as it was originally for TAME and is currently for the

This was _not_ the original plan---the plan was to have identifiers indexed
only at the package level, at least until we need a global lookup for
something else---but that plan was upended by how externs are currently
handled.  So, for now, we need a global scope.

(Externs are resolved by the linker in such a way that _any_ package that
happens to be imported transitively may resolve the import.  This is a
global environment, which I had hoped to get rid of, and which will need to
eventually go away (possibly along with externs) to support loading multiple
programs into the graph simultaneously for cross-program analysis.)

This commit renames the base state for `AirAggregate` to emphasize the fact,
especially when observing it in the `AirStack`, and changes
`AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of
the stack upward, rather than reverse, to prove that the system still
operates correctly with this change in place.

The reason for this direction change is to simplify lookup in the most
general case of non-local identifiers, which are almost all of them in
practice---they'll be immediately resolved at the root once they're
indexed.  This can be done because I determined that I will _not_ support
shadowing; rationale for that will come later, but TAME is intended to be a
language suitable for non-programmer audiences as well.  Note that
identifiers will be resolved lexically within templates in TAMER, unlike
TAME, which means that the expansion context will _not_ be considered when
checking for shadowing, so templates will still be able to compose without a
problem so long as they do not shadow in their definition context.  (I'll
have to consider how that affects template-generating templates later on,
but that's an ambiguous construction in TAME today anyway.)

This _does not_ yet index anything at the root where it wasn't already being
indexed explicitly.

2023-05-10 14:43:33 -04:00
Mike Gerwitz dd6a6dd196 tamer: asg::air::ir::AirPkg::PkgStart: Require name
This requires the name as part of the package definition, which in turn
removes a state (and all the combinations resulting from it) from
AirAggregate, which results in significant complexity reduction for a very
complex part of the system.

Pushing this complexity outward results in a reduction of overall
complexity, and obviates the question of where NIR will receive a generated

2023-05-10 13:57:45 -04:00
Mike Gerwitz 7a6aef00b2 tamer: nir::air::NirToAir: Note about intent to refactor
The comment speaks for itself.

My concern is that this will be especially off-putting to people looking at
TAMER and wondering how one could possibly work with this system.

2023-05-10 13:57:41 -04:00
Mike Gerwitz 4510de38ed tamer: asg::air::pkg: Extract AirPkgAggregate from AirAggregate
This is something I've wanted to do for some time, but the system is
becoming hard enough to reason about (with some attempted future changes)
that I require the consistency afforded by this change.

It's not entirely done---as noted by the TODO for `UnnamedPkg`---but it's
close, and then `AirAggregate` will just be a delegating superstate, like

Importantly, this also puts a package parser on the stack, which will work
better with the stack-based scoping system being developed.  It will also
make it easier to fall back to a base case that I had really wanted to
avoid, and will have more information on in the future: root indexing for a
shared global environment for package-level identifiers.  (Imports are still
package-scoped, but only in appearance, by contributing to the global
environment of the compilation unit during import.  Well, it doesn't do that
yet.  The XSLT compiler works in that way.)

2023-05-10 13:57:39 -04:00
Mike Gerwitz ebdae9ac38 tamer: ld::xmle::lower: Sort only rooted Idents
This is one of many changes that have been lingering that I need to start to
break apart in an attempt to commit the confusing and disappointing
conclusion to this package loading madness.

More information to come.

2023-05-09 15:20:39 -04:00
Mike Gerwitz 5f275fb801 tamer: asg::air::air: Eliminate todo! for unexpected AirIdent
I had apparently forgotten about this, because I didn't benefit from the
exhaustiveness check; this needs to be eliminated so that this doesn't
happen again, and to provide a proper non-panicking error.

2023-05-09 12:35:06 -04:00
Mike Gerwitz 4ec4857360 Revert "tamer: asg::air::ir::AirBind::RefIdent: New optional canonical name"
This reverts commit da7fe96254e425bc7b75f8cf454465b71e27e372.

I'm a fool---this would be pursuant to a future plan that removes AirIdent
opaque tokens.  But for now, I need it on IdentDecl and others, which
currently has a `Source` (that I want to go away, as just mentioned), which
contains the same information.

So maybe more to come on this...

2023-05-09 12:35:06 -04:00
Mike Gerwitz 572337505c tamer: asg::air::ir::AirBind::RefIdent: New optional canonical name
This allows for a canonical package name to be optionally provided to
explicitly resolve a reference against, avoiding a lexical lookup.

This change doesn't actually utilize this new value yet; it just
retains BC.  The new argument will be used for the linker, since it already
knows the package that defined an identifier while reading the object file's
symbol table.  It will also be used by tamec for the same purposes while
processing package imports.


-- squashed with --

tamer: asg::air::ir::RefIdent: CanonicalName=SPair

The use of CanonicalName created an asymmetry between RefIdent and
BindIdent.  The hope was to move CanonicalName instantiation outside of AIR
and into NIR, but doing so would be confusing and awkward without doing
something with BindIdent.

I don't have the time to deal with that for now, so let's observe how the
system continues to evolve and see whether hoisting it out makes sense in the
end.  For now, this works just fine and I need to move on with the actual
goal of finishing package imports so that I can expand templates.

2023-05-09 12:35:06 -04:00
Mike Gerwitz 00492ace01 tamer: obj::xmlo: Bind packages to canonical name
NOTE: This fixes the aforementioned commit that caused the linker to
temporarily fail (670c5d3a5d at time of
writing).  This does introduce an extra forward slash into
`l:dep/preproc:sym/@src`, but that does not appear to cause any
problems.  That will eventually go away, so I'm not going to bother with it
any further.

As the `xmlo` file is lowered into AIR, the name will be prefixed with a
leading slash (if necessary, which it is atm) and will emit an

This means that packages will be properly indexed by their canonical name on
load, which will be important when we share this with tamec.

2023-05-09 12:34:12 -04:00
Mike Gerwitz 13bac8382f tamer: obj::xmlo::{air,reader}::test: Format test cases
Simple reformatting that's consistent with other more recent tests, before I
go making changes.

2023-05-05 10:26:58 -04:00
Mike Gerwitz 48bcb0cdab tamer: asg: Integrate package CanonicalName
This change requires every package to have a canonical name, and performs
namespec canonicalization on imports.

Since all package names are canonicalized, this opens the door to being able
to index package names at import, allowing the object to be shared on the
graph and properly reference a package after it has been resolved.

Note that the system tests' canonicalization is relative to the hard-coded
`/TODO` presently; that will change in the near future once `tamec`
generates names from the provided path.

2023-05-05 10:26:58 -04:00
Mike Gerwitz a9d0f43684 tamer: src::asg::graph::object::pkg::name: New module
This introduces, but does not yet integrate, `CanonicalName`, which not only
represents canonicalized package names, but handles namespec resolution.

The term "namespec" is motivated by Git's use of *spec (e.g. refspec)
referring to various ways of specifying a particular object.  Names look
like paths, and are derived from them, but they _are not paths_.  Their
resolution is a purely lexical operation, and they include a number of
restrictions to simplify their clarity and handling.  I expect them to
evolve more in the future, and I've had ideas to do so for quite some time.

In particular, resolving packages in this way and then loading the from the
filesystem relative to the project root will ensure that
traversing (conceptually) to a parent directory will not operate
unintuitively with symlinks.  The path will always resolve unambigiously.

(With that said, if the symlink is to a shared directory with different
directory structures, that doesn't solve the compilation problem---we'll
have to move object files into a project-specific build directory to handle

Span Slicing
Okay, it's worth commenting on the horridity of the path name slicing that
goes on here.  Care has been taken to ensure that spans will be able to be
properly sliced in all relevant contexts, and there are plenty of words
devoted to that in the documentation committed here.

But there is a more fundamental problem here that I regret not having solved
earlier, because I don't have the time for it right now: while we do have
SPair, it makes no guarantees that the span associated with the corresponding
SymbolId is actually the span that matches the original source lexeme.  In
fact, it's often not.

This is a problem when we want to slice up a symbol in an SPair and produce
a sensible span.  If it _is_ a source lexeme with its original span, that's
no problem.  But if it's _not_, then the two are not in sync, and slicing up
the span won't produce something that actually makes sense to the user.  Or,
worse (or maybe it's not worse?), it may cause a panic if the slicing is out
of bounds.

The solution in the future might be to store explicitly the state of an
SPair, or call it Lexeme, or something, so that we know the conditions under
which slicing is safe.  If I ever have time for that in this project.

But the result of the lack of a proper abstraction really shows here: this
is some of the most confusing code in TAMER, and it's really not doing
anything all that complicated.  It is disproportionately confusing.

2023-05-05 10:26:56 -04:00
Mike Gerwitz e3a68aaf9e tamer: span: Introduce rslice, slice_{head,tail}
These are used by an upcoming commit, where I'll have much more to say about
the topic of span slicing.

2023-05-05 10:25:35 -04:00
Mike Gerwitz 670c5d3a5d tamer: asg::graph: Require name for non-imports
NOTE: This temporarily breaks `tameld`.  It is fixed in a future commit when
names are bound.  This was an oversight when breaking apart changes into
separate commits, because the linker does not yet have system tests like
tamec does.

This is preparing for a full transition to requiring a canonical package
name.  The previous `Unnamed` variant has been removed and `AirAggregate`
will provide a default `WS_EMPTY` name, as `Pkg` had done before.

The intent of this change is to allow for consulting the index before a
new `Pkg` object is created on the graph, but we're not quite ready for that

Well, that's not entirely true---the linker can be ready for that.  But the
compiler needs to canonicalize import paths relative to the active package
canonical name, which it can't even do yet because tamec isn't generating a

So maybe the linker will be first; it's useful to have that in a separate
commit anyway to emphasize the change.

2023-05-05 10:24:47 -04:00
Mike Gerwitz 799f2c6d96 tamer: tameld: Produce first error
...this has apparently been consuming errors for some time.  This would
cause the parser to enter an invalid state in some cases and terminate.

This would _not_ permit an invalid link, as the graph would not be correct,
but it was masking the actual error.

This part of linker is in dire need of tests.  This also ought to be
replaced with tamec's approach of reporting all errors.

2023-05-04 16:04:52 -04:00
Mike Gerwitz 7cfe6a6f8d tamer: asg::graph: Index Root->Pkg with canonical names
The previous commit introduced canonical names, and this uses them to index.

The next step will be to utilize those names to look up packages on
definition rather than creating a new package node, so that references to
yet-to-be-defined (or yet-to-be-imported) packages can be resolved on the

2023-05-02 16:15:07 -04:00
Mike Gerwitz 92c9c9ba2f tamer: asg: Introduce package canonical name concept
This is already a concept in the XSLT-based compiler, where each package has
a `package/@name` generated from its path.  The same will happen with tamec.

Before we can load packages into the graph, we need canonical identifiers so
that they can be indexed.  The next commit will handle indexing using this

2023-05-02 16:08:39 -04:00
Mike Gerwitz 56ab671363 tamer: nir::interp: Ignore Text tokens
The documentation explains the intent here---existing LaTeX documentation.

The intent was to simply copy the documentation into a LaTeX document based
on the lvspec package that I had created long ago.  Of course, that's not
appropriate---we're a DSL and should provide first-class support for
documentation that will compile properly into the target format, whether it
be LaTeX, HTML, JS, or anything else.

2023-04-30 15:06:58 -04:00
Mike Gerwitz 068804b397 tamer: Remove {ret}map:___{head,tail}
These have been a pain in the ass since TAMER began.

It seemed like a good idea at the time to have static code generated in this
way, but the lack of explicit dependencies just makes this a mess and works
against the operating theory of the system.

Furthermore, the _same_ static fragments were generated for each and every
map package.

There is still a post-link step (standalones) handled in XSLT; the
previously-static code has been moved there.  This will eventually be
integrated into tameld itself, once TAMER has facilities for JS generation.

(This was discovered while trying to parent identifiers to packages.)

2023-04-30 15:06:47 -04:00
Mike Gerwitz 77ada079e1 tamer: asg::graph::Asg.graph: Finally encapsulate
With the previous commit using a visitor implemented within the `asg`
module, we can now finally encapsulate the graph.  This is a wonderfully
liberating, long-awaited change, since I have been fighting with the lack of
encapsulation for some time; it has made certain changes challenging and has
made the system more difficult to reason about.  It also made it impossible
to assert that invariants were _actually_ properly enforced, if things could
just peer into and modify the graph directly, out from underneath the API
that provides those assurances.

This also removes our dependency on Petgraph outside of the `asg`
module.  There are no plans to migrate away from it currently; we'll see how
the graph continues to evolve over time and what redundancies are introduced
with our data structures.  It may render petgraph unnecessary.

Interestingly, because my DFS implementation is so similar to Petgraph's,
the emitted ordering is _identical_ between this commit and the previous.

2023-04-28 15:36:07 -04:00
Mike Gerwitz 78c1a9136e tamer: ld::xmle::lower: Use asg::graph::visit::topo::topo_sort
This integrates the new topological sort, replacing the previous
implementation in the linker.

This will now allow encapsulating the graph, finally, and ensures that
future changes can be fully maintained within the `asg` module.

More cleanup will come over time.

2023-04-28 15:26:47 -04:00
Mike Gerwitz 9b53a5e176 tamer: asg::graph::visit::topo: Cut cycles
This commit includes plenty of documentation, so you should look there.

It's desirable to describe the sorting that TAME performs as a topological
sort, since that's the end result we want.  This uses the ontology to
determine what to do to the graph when a cycle is encountered.  So
technically we're sorting a graph with cycles, but you can equivalently view
this as first transforming the graph to cut all cycles and then sorting it.

For the sake of trivia, the term "cut" is used for two reasons: (1) it's an
intuitive visualization, and (2) the term "cut" has precedence in logic
programming (e.g. Prolog), where it (`!`) is used to prevent
backtracking.  We're also preventing backtracking, via a back edge, which
would produce a cycle.

2023-04-28 14:33:48 -04:00
Mike Gerwitz c2c1434afe tamer: asg::graph::visit::topo: Cycle detection
This introduces cycle detection, but it does not yet filter ontologically
permitted cycles, which will be needed prior to utilizing this in `tameld`.

There's a considerable amount of documentation here.  While the
implementation is fairly simple, there are important algorithmic decisions,
both in the DFS construction and the derivation of the cycle path from data
that already exists.

This also supports recovery (by ignoring cycles), which can then be utilized
to find more cycles and other errors in the system.

2023-04-27 16:28:57 -04:00
Mike Gerwitz e3094e0bad tamer: asg::graph::visit::topo: Introduce topological sort
This is an initial implementation that does not yet produce errors on
cycles.  Documentation is not yet complete.

The implementation is fairly basic, and similar to Petgraph's DFS.

A terminology note: the DFS will be ontology-aware (or at least aware of
edge metadata) to avoid traversing edges that would introduce cycles in
situations where they are permitted, which effectively performs a
topological sort on an implicitly _filtered_ graph.

This will end up replacing ld::xmle::lower::sort.

2023-04-26 09:51:45 -04:00
Mike Gerwitz be05fbb833 tamer: asg::graph::visit{=>::ontree}: Move into submodule
This reorganization makes way for more traversals.

2023-04-24 13:51:04 -04:00
Mike Gerwitz 42aa5bd407 tamer: asg::graph: Root->Ident {tree=>cross} edge
tameld isn't yet adding edges to Idents from their associated Pkg (see
previous commit), but this formalizes how the ontology will interpret such a
relationship.  The idea is that Idents are always owned by Pkgs, but they
may be optionally explicitly rooted, which will be used by a particular type
of DFS walk that is about to be written, which can ignore Root->Pkg and
focus instead on cross edges to Idents.

Though it's not lost on me that now that I'll be introducing a DFS for the
linker, the terms "cross" and "tree" edge now become ambiguous; I used to
call them "ontological X edge", but I had fallen out of that habit; perhaps
I need to reintroduce that rigor.

2023-04-24 09:44:02 -04:00
Mike Gerwitz 48d9bca3b7 tamer: obj::xmlo: Add Pkg nodes for identifiers
This modifies the xmlo reader, xmlo->AIR lowering, and AIR->ASG to introduce
a package for identifiers.  It does not yet, however, add edges from the
package to the identifier.

Once edges are added, the DFS will change in undesirable ways, which will
require a new implementation.  This is desirable to decouple from Petgraph
anyway, and then will be able to restore the prior single-pass sort+cycle

That will also encapsulate visiting behavior within the `asg::graph` module
and, in turn, allow encapsulating `Asg.graph` finally.

2023-04-21 16:24:11 -04:00
Mike Gerwitz 34dad122fd tamer: asg::air::test: Remove `Air` and `Parsed` enum prefixes from variants
Same spirit as previous commits.  This is committed separately from the
changes that follow to eliminate its distraction.

2023-04-21 15:37:27 -04:00
Mike Gerwitz 1f371b6ba6 tamer: obj::xmlo::air: More explicit dead states
This doesn't go far enough, but it elaborates a bit---the existing was far
too much of a catch-all.  It's important to take advantage of exhaustiveness
checks to ensure each transition is properly accounted for.

This parser is going to get more work over time, including right now, so I'm
not going to go too deep into this yet, but it's be useful (as a reader) to
compare it to e.g. asg::air's parsers' explicit enumeration of states and
favoring of explicit errors over dead state transitions.

2023-04-21 11:48:56 -04:00
Mike Gerwitz dd47fc564d tamer: obj::xmlo: More concise identifiers
This follows conventions of other, more recently written, systems.

2023-04-21 11:48:55 -04:00
Mike Gerwitz e13817c203 tamer: obj::xmlo::air::test: Extract into own file 2023-04-20 16:46:32 -04:00
Mike Gerwitz 6f68292df5 tamer: asg::graph::{index_identifier=>index}: Generalize
This may now index _any_ type of object, in preparation for indexing package
import paths.  In practice, this only makes sense (at least currently) for
`Pkg` and `Ident`.

This generalization also applies to `Asg::lookup_or_missing`.

2023-04-20 16:46:30 -04:00
Mike Gerwitz f183600c3a tamer: asg: Move Ident-specific methods off of Asg
Historically, the ASG was better described as a "dependency graph",
containing only identifiers (which are simply called "symbols" in the
XSLT-based compiler).  Consequently, it was appropriate for the graph to
have operations specific to identifiers.  (Indeed, that's the only type of
object the graph supported.)

Much has changed since then.  This cleans things up, and makes parenting
identifiers to root an _explicit_ operation.  This will make it easier to
move forward with handling of scope, and importing identifiers into
packages, and removing `Source`, and so on.

2023-04-19 12:40:35 -04:00
Mike Gerwitz 46551ee298 tamer: ld::xmle::lower::test: Extract into own file
2023-04-19 12:40:35 -04:00
Mike Gerwitz 778e90c81d tamer: asg::air: Index package identifiers on `Pkg` rather than `Root`
I've been torturing myself trying to figure out how I want to generalize
indexing, lookups, and value numbering in a way that is appropriate for this
project (that is, not over-engineered relative to my needs).

Before I can do much of anything, though, I need to stop having indexing
only as a `Root` thing (previously it wasn't even tied to `Root`).  This
makes that change for tamec, but temporarily removes scoping concerns until
I can add more specific types of indexing.

Not only does this allow cleaning up some `Ident`-specific stuff from `Asg`,
but the cleanup also helps to show that portions of the system aren't still
using Root-based globals.

The linker (`tameld`) still uses the old `global` methods for now; those
will eventually go away, but this needs to change to unify both tamec and
tameld once we get to imports as part of the compiler.

2023-04-19 12:40:34 -04:00
Mike Gerwitz c367666d8e tamer: nir::tplshort: Generate @desc for generated template
This is required by the XSLT-based compiler, since the `xmli` we're
producing acts as a new source file.

2023-04-13 09:30:27 -04:00
Mike Gerwitz 590c4b7b06 tamer: NIR->xmli: Support template/@desc
This is needed to then support `@desc` for shorthand desugaring; it's
required by the XSLT-based compiler (and will eventually be required by
TAMER too).

2023-04-12 15:53:16 -04:00
Mike Gerwitz 5dd77e7b41 tame: rater.xsd: templateName: Permit multiple leading/trailing underscores
This is needed by TAMER's template desugaring.  The XSD is superceded by
`nir::parse`, but can't go away until TAMER fully supplants the XSLT-based

...and after all this time, I still never got rid of the duplicate XSD.  Or
even recall which one is the duplicate.

2023-04-12 14:54:00 -04:00
Mike Gerwitz 2325eb1b2f tame: preproc/template.xsl: param-copy: Utilize TAMER application convention
TAMER desugars shorthand template application bodies (`@values@`) into _the
name of a closed template_ whose body should be expanded into place.  This
change recognizes that convention, and makes use of it.

Desugaring is part of `nir::tplshort`.

2023-04-12 14:52:06 -04:00
Mike Gerwitz b7aae207c2 tamer: Rust v1.{68=>70}: Stabalized nonzero_min_max and is_some_and
These two features have been stabalized in Rust 1.70.
2023-04-12 12:04:13 -04:00
Mike Gerwitz af43e35567 tamer: nir::air: Reject Todo* tokens
XIRF->Nir produces `Todo` and `TodoAttr` tokens for many different
things.  The previous approach was to ignore those things so that I could
begin adding portions of packages to the graph and observe how that goes.

But now that I'm starting to be able to compile certain packages that
utilize only small subsets of TAME features, I need to have confidence that
I'm fully parsing them.  This means rejecting tokens that I haven't yet
gotten to.

2023-04-12 12:04:13 -04:00
Mike Gerwitz e88800af42 tamer: asg: Basic `Doc::Text` support
This supports arbitrary documentation as sibling text (mixed content, in XML
terms).  The motivation behind this change is to permit existing system
tests to succeed when `Todo | TodoAttr` are both rejected, rather than
having to ignore this.

TAME has always had a philosophy of literate documentation, however it was
never fully realized.  This just maintains the status quo; the text is
unstructured, and maybe will be parsed in the future.

Unfortunately, this does _not_ include the output in the `xmli` file or the
system tests.  The reason has nothing to do with TAMER---`xmllint` does not
format the output when there is mixed content, it seems, and I need to move
on for now; I'll consider my options in the future.  But, it's available on
the graph and ready to go.

2023-04-12 12:04:12 -04:00
Mike Gerwitz 647e0ccbbd tamer: Re-introduce literal parsing for xmlns in NIR for PackageStmt
This _only_ re-introduces for PackageStmt since that's all I have tests for
at present.  More will be re-added later.

They were previously removed when the attribute parsing was upended in

This does lose the attribute name, compared to before; that'll ideally be
re-added, and I'll explore options for doing so later, since I also want
them in other contexts.  But it needs to be done generically (not

This had to be done before blowing up on TODOs, or system tests would fail.

2023-04-12 11:59:49 -04:00
Mike Gerwitz acafe91ab9 tamer: nir::Nir::Todo: Add Span
This is in preparation for throwing errors (with diagnostic information) on
yet-to-be-supported tokens, so that I can confidently compile individual
packages without worrying that something is just being ignored.

This makes obvious that `ele_parse!` had a different design in mind
previously, and it's now resulting in a lot of boilerplate; I'll address
that in the future once I'm certain requirements have been settled on, since
I've spent far too much time on it to waste more.

2023-04-12 11:59:49 -04:00
Mike Gerwitz 9cb6195046 tamer: asg: Add basic Doc support (for @desc)
This introduces a new `Doc` object that can be owned by `Expr` (only atm)
and contain what it describes as a concise independent clause.  This
construction is not enforced, and is only really obvious today via the
Summary Pages.

There's a lot of latent and unrealized potential in TAME's documentation
philosophy that was never realized, so this will certainly evolve over
time.  But for now, the primary purpose was to get `@desc` working on things
like classifications so that `xmli` output can compile for certain

2023-04-12 11:59:48 -04:00
Mike Gerwitz 0163391498 tamer: asg::graph::object::prelude: New module to reduce imports
These are used by virtually every `ObjectKind`; I've been meaning to do this
for a while, but now that I'm about to introduce a new one (`Doc`), let's
just get it out of the way.

2023-04-07 09:56:50 -04:00
Mike Gerwitz f4653790da tamer: NIR->xmli: Represent package imports
This doesn't do the actual hard work yet of resolving and loading a package,
but it does place it on the graph and re-derive it into the xmli output.

2023-04-07 09:44:16 -04:00
Mike Gerwitz 82e228009d tamer: NIR->xmli: Basic match support
This introduces `<match on="foo" />` and `<match on="foo" value="bar" />`,
which are both equality predicates.  Other types of predicates are not yet

This change is a bit messy and leaves a bit to be desired.  `NirToAir` is
quite messy and needs some cleanup.  There's also the issue of introducing
XML-specific errors in NIR so that users know what things like "subject"
mean, but not being able to do so yet because NIR is agnostic to the source
document type; another layer of abstraction is needed.

But, my priority is first to get derivation of a particularly
expensive (generated) package in our internal systems working first.

2023-04-06 22:40:18 -04:00
Mike Gerwitz 1f2ead7f9b tamer: nir: Introduce disambiguating RefSubject
The alternative I was floating was a tagged `Ref` (that is, with an enum
within it), but I settled on this for now, in part for a more concise
notation with the mapping in nir::parse.

We'll see how this evolves.  For now, it's not important with the only thing
that uses ref in nir::parse, which is template application.

This was introduced for `match`, which is to come shortly.

2023-04-06 10:28:27 -04:00
Mike Gerwitz 7b2acb65c5 tamer: nir::air::test: Formatting and enum prefix elision
This just makes easier to read and more concise.  I'm about to add a number
of tests and the verbosity was off-putting.

2023-04-06 09:33:44 -04:00
Mike Gerwitz e8371c452e tamer: Remove wip-nir-to-air feature flag in favor of existing wip-asg-derived-xmli
The latter has always enabled the former, and there's really no reason I'd
enable one but not the other at this point.  It's just confusing.

2023-04-05 22:28:30 -04:00
Mike Gerwitz c0e5b1d750 tamer: asg::air: Template application within expressions
This recognizes template application within expressions.  Since expressions
can occur within templates, this can occur arbitrarily deeply.

And with that, we have the core of the template system represented on the
graph.  Of course, there are some glaring scoping issues to be resolved, but
those aren't unique to template application.

2023-04-05 15:49:25 -04:00
Mike Gerwitz daa8c6967b tamer: asg: Initial nested template supported
I had hoped this would be considerably easier to implement, but there are
some confounding factors.

First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file.  But to do so
successfully, some assumptions had to be made.

The primary issue is that of scope.  The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations.  In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort.  The equivalent here would
be a graph traversal, which is not ideal.

I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.

2023-04-05 15:46:44 -04:00
Mike Gerwitz a738a05461 tamer: asg::graph::object::rel: Hash impls for ObjectIndexTo{,Tree}
All ObjectIndex-like objects hash using only the underlying identifier,
which ultimately boils down to a `NodeIndex` (petgraph), which is just a
u32.  And so in that sense, the only purpose we have for hashing it is to
(a) reduce the space required to store mappings, and (b) compose with other

2023-04-05 15:46:42 -04:00
Mike Gerwitz 3660c15d5a tamer: asg::graph::rel::ObjectIndexTreeRelTo: New trait and related
This creates another trait and struct `ObjectIndexToTree` that assert a
stronger invariant than `ObjectIndexRelTo`---that not only does it uphold
the invariants of `ObjectIndexRelTo`, but also that it represents a _tree_
edge, which indicates _ownership_ rather than just a reference.

This will be used to statically infer what can serve as a scope boundary for
upcoming changes.  Specifically, anything that can own an `Ident` introduces
a new level of scope.

2023-04-04 14:33:34 -04:00
Mike Gerwitz f1495f8cf4 tamer: asg::graph::object: Move `lookup_local_linear` to `ObjectIndexRelTo`
This allows this method to be used on anything that is able to relate to an
identifier, which is needed for the changes being made for the template

This linear lookup is actually going away (as hinted at by preceding
commits); this is extracted as part of a larger change and I wanted to get
it committed to make it easier to follow upcoming changes.

2023-04-03 16:14:31 -04:00
Mike Gerwitz 02dba0d63a tamer: asg::graph::Asg: Index by (SymbolId, NodeIndex) pair
The prior commit begins to explain the end goal of being able to index
identifiers outside of the global environment.

This change continues to index things as before, but introduces a new key
based on the pair of the symbol id together with a node that is _part of_
its target environment.  The only environment utilized at the moment (in this
commit) is that of the root node (which is the global scope), in both
indexing and lookup.  Future commits will extend this, and contain more
information about and rationale for the implementation.

The new general index methods are restricted to `pub(super)` until an
abstraction can be put in place that is responsible for environment
indexing; that's a responsibility that is currently handled by
`AirAggregateCtx` for tamec, and the linker has no scoping
requirements since all of that has already been dealt with.

2023-04-03 16:14:30 -04:00
Mike Gerwitz 5b0a4561a2 Revert "Revert "tamer: asg::graph::index: Use FxHashMap in place of Vec""
This reverts commit 1b7eac337cd5909c01ede3a5b3fba577898d5961.

This is a revert of the previous revert, just so that I (and you) have
references to prior rationale.

This was previously reverted because it wasn't worth doing, but now we have
a situation where we need to begin implementing lexical scoping rules for
nested containers (packages and templates).  In particular, as you'll see in
the commits that follow, we need to be able to look up an identifier that
may have been created as Missing at one level of scope (certain types of
blocks), but then define it at another level.

Or, even more simply at this point, since I'm not yet doing anything
sophisticated with scope: we're only indexing in the global environment, and
we need to be able to index elsewhere too.

The next commit will go into more information, but suffice it to say for now
that indexing is going to get more complicated than a SymbolId.

Sticking with FxHash for now; we don't need a stable hash now.

2023-04-03 15:15:54 -04:00
Mike Gerwitz 6d35e8776c tamer: asg::air: InvalidExpansionContext in place of TODO
There are no such invalid expansion contexts yet, but this gets rid of the
final remaining TODO from introducing the stack.  With the existing feature
set, at least.

2023-03-31 14:23:26 -04:00
Mike Gerwitz e3d60750a9 tamer: asg::air: Errors for rooting_ci() TODOs
This eliminates the TODOs that existed when looking for an OI for rooting an

The change to `rooting_ci` is ridiculous, but I want to get other things
done before I jump down the rabbit hole of generalizing that (indexing local
identifiers).  Though I have an approach in mind.

2023-03-31 13:57:11 -04:00
Mike Gerwitz a33d0c4ea5 tamer: asg::air: Consolate nested PkgStart
Just some continued cleanup.

Unfortunately, we have sacrificed knowing a package OI must exist
statically, even though one will always be available.

2023-03-30 22:28:22 -04:00
Mike Gerwitz 0e0b72ff5f tamer: asg::air: Generalize control transfer convention
The diff should make this refactoring obvious.  The provided documentation
explains why it operates the way that it does.

2023-03-30 16:38:03 -04:00
Mike Gerwitz 558f1c96b1 tamer: asg::air: Extra AirExpr parsing from AirTplAggregate
This has AirAggregate preempt Expr parsing in the same way as templates,
rather than having `AirTplAggregate` concern itself with expression
tokens.  This continues to simplify `AirTplAggregate`, which was getting
quite complex not too long ago.

A pattern is now emerging for the call/ret convention for preemption.  That
was intentional, but it's nice to see it manifest so obviously before I
abstract it away.

2023-03-30 15:44:14 -04:00
Mike Gerwitz f29e3cfce1 tamer: asg::air: Use StateStack
This was extracted from xir::parse::ele in previous commits.  The
conventions help to ensure that pushing and returns are being performed
correctly.  The abstraction will continue to evolve.

This ends up using `Ready` as the dead state.  I need to determine if this
is ideal, and if so, maybe just use `Default`, otherwise yield an error.

2023-03-30 15:44:14 -04:00
Mike Gerwitz e6c6028b37 tamer: xir::parse::ele: Move StateStack into parse::state
This will be utilized by `AirAggregate`.

2023-03-30 15:44:12 -04:00
Mike Gerwitz 11a4fdfb26 tamer: xir::parse::ele::StateStack: {Array=>}Vec
The use of ArrayVec doesn't buy us anything anymore.  There is no difference
in performance through my own benchmarking (at least on our systems), and
the game has changed since this was written: the size of the states is much
smaller since we're no longer aggregating attributes.  Further, the use of
ArrayVec during development was also to keep memory allocation away from
various parts of the code, which simplified analysis of the binary that was
produced.  Maybe it also reduced memory contention, but clearly that has no
observable impact.

The use of `Vec` removes the arbitrary bound, though I still kept one around
just in case something goes wrong, so TAMER will terminate.  Even though the
token stream is bounded in size, lookahead does create recursion, and the
system cannot (as written) prove that it doesn't.

This is preparing for extracting `StateStack` into `parse` for use with

2023-03-30 10:17:15 -04:00
Mike Gerwitz d091103983 tamer: asg::air::tpl: Remove Expr delegation (move to parent)
`AirAggregate` now handles all delegation to `AirExprAggregate`.  This is
possible because `AirAggregate` is now the superstate for each of these
parsers, so `AirTplAggregate` is able to transition to a state that is not
its own.

This does not go so far as reaching the ultimate objective---having nested
template support---even though it'd be fairly simple to do now; there's
going to be a number of interesting consequences to these changes, and a bit
of cleanup is still needed, and I want tests observing this functionality to
accompany those changes.  That is: let's keep this a refactoring, to the
extent that it's possible.

Things are getting much easier to understand now, and much cleaner.

2023-03-30 09:26:11 -04:00
Mike Gerwitz c59b92370c tamer: parse::state: Superstate support for Token type lifting
What hell have I gotten myself into.

In the end, this wasn't too bad, but the initial batch of errors was really
demotivating; the diff does this no justice.  `Lookahead::into_super` was
created to help tame those errors. I can move forward.  Imagine my disappointment when I ran into this
when expecting from previous work that superstates would now work properly
for the AirAggregate parsers.

(The reason this was needed is because AirAggregate splits tokens into
subtypes for child parsers.)

2023-03-29 15:50:06 -04:00
Mike Gerwitz 68e2d5d10e tamer: parse::util::expand: Delete module
Oh, boy, I had forgotten about this, until I started working on some
SuperState stuff and discovered this again due to a compiler error.  Don't
want to fix something that isn't used.

But this does not bring back great memories.  It's unfortunate that it
didn't work out; I'm pretty sure this was part of ~1mo of wasted effort
going down a path that I ultimately had to abort.  Not good times.  I'm
still behind from it.

2023-03-29 15:30:51 -04:00
Mike Gerwitz 15fd2de437 tamer: asg::air::expr: Eliminate RootStrategy
I love deleting code I just wrote...

This doesn't solve the underlying problems with identifiers, but it does at
least lift it into the `AirAggregateCtx`, allowing `AirExprAggregate` to be
even further simplified.  Now the `From` implementation is not specialized
and we can readily convert to a SuperState.

There's still a lot of TODOs here, though.  And some of them will
unfortunately require runtime checks where there was previously a
compile-time check.  But that's okay in a lot of the cases, because the
empty behavior will replace existing error checks.

2023-03-29 13:49:05 -04:00
Mike Gerwitz 26ddb2ae9d tamer: asg::air::expr: Remove RootStrategy::hold_dangling
Whether or not dangling expressions are permitted is now based solely off of
the stack context, which is also much more intuitive.

`RootStrategy` now only does one thing, and the existing comments describe
why it exists despite that one thing seeming very similar.

`RootStrategy` further alludes to how `ExprStack` could also be
eliminated, should it be worth doing so.  It is a tad redundant now with the
new stack.

2023-03-29 13:02:01 -04:00
Mike Gerwitz 525adb8a6c tamer: asg::air: Eliminate parent context from AirExprAggregate
This does the same thing to `AirExprAggregate` that was previously done for
`AirAggregate`, taking all parent context from the stack.

This results in a fairly significant simplification of the code, which is
nice, and it makes the `RootStrategy` obviously obsolete in the dangling
case, which will result in more refactoring to simplify it even more.

I regret not taking this route to begin with, but not only was I hoping I
wouldn't need to, but I was still deriving the graph structure and wasn't
sure how this would eventually turn out.  These commits serve as a proof of
necessity.  Or, at least, concrete rationale.

It's worth noting that this also introduces `From` implementations for
`AirAggregate` and the child parsers, and then uses _that_ to push context
from the `AirTplAggregate` parser.  This means that we're just about ready
for it to serve as a superstate.  But there is still a specialization of
`AirExprAggregate` in that `From` impl, which must be removed.

2023-03-29 13:02:00 -04:00
Mike Gerwitz 755c91e04a tamer: asg::air: Merge AirStack into AirAggregateCtx
Having an extra layer of abstraction was inconvenient, and unnecessary.

2023-03-29 12:58:36 -04:00
Mike Gerwitz a5b4eda369 tamer: asg::air::AirAggregate: Remove Pkg context from child parser states
This is more of the same of the previous commit, but in a more digestable
chunk.  We now have child states that are able to be constructed using a
simple `From`, which is important to making `AirAggregate` a `SuperState`.

This also makes `AirStack` act like a prototype chain for `ObjectIndex`es,
creating environments where context shadows.  The linear search should only
have to check the last two frames (e.g. an Expr has a parent Pkg or Tpl
context which will have a `rooting_oi` value), and this is only done during
a rooting operation.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 1ef1290ee9 tamer: asg::air: Begin to derive context from stack
This begins to introduce `AirStack` and starts to migrate context away from
the individual `ParseState`s onto the stack.

I should have started to commit earlier; this is getting a bit large and
makes it hard to follow what I'm doing so, hopefully stopping a little bit
short will allow the following commit to show that.

This is a work-in-progress change.  All tests pass, but the refactoring is
incomplete.  The `AirStack` abstraction is _also_ incomplete and will have
better, more domain-specific operations that make it harder to mess up
pairing pushes with pops.

The purpose of doing this is to allow `AirAggregate` to serve exclusively as
a sum state, which can then become a SuperState, much like `ele_parse!`'s

The _end_ goal of all of this is arbitrary template nesting.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 2ae33a1dfa tamer: asg::graph::object: ObjectIndexTo and ObjectIndexRelTo
The graph's ontology is defined in the direction of the edge: from OA
to OB.  This is enforced by the type system to ensure that no code path is
able to generate an invalid graph.

But that also makes it very difficult to work with a generic source to a
specific target.

This introduces a `ObjectIndexRelTo` trait that says whether `Self` is able
to be related to some `ObjectKind` `OB`, implements it for `ObjectIndex
where ObjectRelTo<OB>`, and introduces a new semi-opaque type
`ObjectIndexTo` that allows for the source `ObjectIndex` to be generic.

This then redefines some existing graph primitives in terms of
`ObjectIndexRelTo`, in particular creating edges, so that `ObjectIndex` can
be used as today, and the new `ObjectIndexTo` can be used in the same way
with the same API, without violating the graph ontology.

This will be used by `AirAggregate` to create dynamic targets for rooting
and splicing/expansion.

2023-03-29 12:58:35 -04:00
Mike Gerwitz eebacb52cc tamer: asg::air::AirAggregate: Remove waiting AirExprAggregate
To simplify things in support of upcoming changes, we'll just instantiate a
new one as needed.  This doesn't have an appreciable performance impact, so
the optimization is premature.  It was done just because it was more of the
same that TAMER was already doing, but now it's making things more

2023-03-29 12:58:35 -04:00
Mike Gerwitz b1ce7aaf29 tamer: asg::air: AirAggregateCtx: New AirAggregate::Context
Future changes to `AirAggregate` are going to require additional context (a
stack, specifically), but the `Context` is currently utilized
by `Asg`.  This introduces a layer of abstraction that will allow us to add
the stack.

Alongside these changes, `ParseState` has been augmented with a `PubContext`
type that is utilized on public APIs, both maintaining BC with existing code
and keeping these implementation details encapsulated.

This does make a bit of a mess of the internal implementation, though, with
`asg_mut()` sprinkled about, so maybe the next commit can clean that up a
bit.  EDIT: After adding `AsMut` to a bunch of asg::graph::object::*
methods, I decided against it, because it messes with the inferred
ownership, requiring explicit borrows via `as_mut()` where they were not
required before.  I think the existing code is easier to reason about than
what would otherwise result from having `mut asg: impl AsMut<Asg>`

2023-03-29 12:58:35 -04:00
Mike Gerwitz fc569f7551 tamer: asg::air::tpl: Distinct, generalized root and targets
Previously, `AirTplAggregate` worked only in a `Pkg` context, being able to
root `Tpl` `Ident`s in `Pkg` and expand only into `Pkg`.  This still does
the same, but generalizes to allow for different roots and expansion

This will be utilized to parse nested templates.

2023-03-29 12:58:35 -04:00
Mike Gerwitz e1c8e371d5 tamer: nir::tplshort: Desugar nested template applications
I'm happy with how this ended up turning out---I was able to accomplish this
without having to introduce any additional state to the parser (I _removed_
a state, actually) by tweaking NIR a bit in a previous commit.

We can't update the system test yet, though, because nested templates are
not yet supported by asg::air::tpl; that'll come next.  If you try, you'll
be greeted with this error presently (which is worth showing since you'll
never see it unless you're hacking TAMER):

,=====[ ./tests/xmli/template/ logs ]======
| thread 'main' panicked at 'not yet implemented: internal error:
| note: nested tpl open
|    --> ./tests/xmli/template/src.xml:129:5
|     |
| 129 |     <t:inner-short />
|     |     -------------- note: for this template
| !!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!
| !!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!
| !!! This message means that TAMER has encountered an      !!!
| !!! unrecoverable error that forced it to terminate       !!!
| !!! processing.                                           !!!
| !!!                                                       !!!
| !!! TAMER has attempted to provide you with contextual    !!!
| !!! information above that might allow you to work around !!!
| !!! this problem until it can be fixed.                   !!!
| !!!                                                       !!!
| !!! Please report this error, including the above         !!!
| !!! diagnostic output beginning with 'internal error:'.   !!!
| !!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ !!!
| ', src/asg/air/
| note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
| Command exited with non-zero status 101
| 0/165fault 0/8io 3528rss 14/2ctx
| /home/[...]/tame/tamer/target/debug/tamec -o ./tests/xmli/template/out.xmli --emit xmlo ./tests/xmli/template/src.xml
`====[ end ./tests/xmli/template/ logs ]====

2023-03-29 12:58:35 -04:00
Mike Gerwitz 6581c9946c tamer: nir::air: Remove Nir and NirEntity enum prefixes from variants
This is a long-overdue change to make this easier to read, but I'm _still_
holding off on refactoring, since there's still a lot of room for different
patterns to form with all of NIR that is left.

2023-03-29 12:58:35 -04:00
Mike Gerwitz e595698309 tamer: nir: Apply*Short variants
This adds explicit variants for shorthand template application.  This is
less cryptic, and we'll be able to check for the close directly during

2023-03-29 12:58:35 -04:00
Mike Gerwitz 975f60bff9 tamer: nir::tplshort: Desugar body into @values@
This represents a significant departure from how the XSLT-based TAME handles
the `@values@` param, but it will end up having the same effect.  It builds
upon prior work, utilizing the fact that referencing a template in TAMER
will expand it.

The problem is this: allowing trees in `Meta` would add yet another
container; we have `Pkg` and `Tpl` already.  This was the same problem with
template application---I didn't want to add support for binding arguments
separately, and so re-used templates themselves, reaching the generalization
I just mentioned above.

`Meta` is intended to be a lexical metasyntatic variable.  That keeps its
implementation quite simple.  But if we start allowing trees, that gets
rather complicated really quickly, and starts to require much more complex
AIR parser state.

But we can accomplish the same behavior by desugaring into an existing
container---a template---and placing the body within it.  Then, in the
future, we'll parse `param-copy` into a simple `Air::RefIdent`, which will
expand the closed template and produce the same result as it does today in
the XSLT-based system.

This leaves open issues of closure (variable binding) in complex scenarios,
such as in templates that introduce metavariables to be utilized by the
body.  That's never a practice I liked, but we'll see how things evolve.

Further, this does not yet handle nested template applications.

But this saved me a ton of work.  Desugaring is much simpler.

The question is going to be how the XSLT-based compiler responds to this for
large packages with thousands of template applications.  I'll have to see
if it's worth the hit at that time, or if we should inline it when
generating the `xmli` file, producing the same `@values@` as
before.  But as it stands at this moment, the output is _not_ compatible
with the current compiler, as it expects `@values@` to be a tree, so a
modification would have to be made there.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 120f5bdfef tamer: nir::tplshort: Remove variant enum prefixes
This just cleans up a little before I introduce more code, making this
easier to read.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 9c0e20e58c tamer: asg: Shorthand and long-form template arguments
This applies to template application only; there's still some work to do for
template parameters in definitions (well, for deriving them in `xmli` at
least).  And, as you can see, there's still a lot of TODO items here.

I ended up backtracking on tree edges to Meta, and even on cross edges to
Meta, because it complicated xmli derivation with no benefit right now;
maybe a cross edge will be re-added in the future, but I need to move on and
see where this takes me.

But, it works.

2023-03-29 12:58:35 -04:00
Mike Gerwitz fcd25d581c tamer: asg::air::expr: Do not cache (globally) identifiers created with StoreDangling
I'm not happy with this implementation.  The linear search is undesirable,
but not too bad (and maybe wouldn't even be worth caching, if this were the
whole story), but we _also_ need to prevent duplicate identifiers.  We are
not going to want to perform a linear search of a linked list (effectively)
every time we add an identifier to check for uniqueness, so I think the
caching is going to have to be generalized very shortly anyway.

As it stands now, a duplicate identifier would cause an error at expansion
time.  That's not what we want, but it's not terrible, because you can have
that same problem in normal circumstances without local conflicts.

But this'll be used for metavariables as well, where we absolutely _do_ want
to fail at template definition time.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 1c7df894ea tamer: asg::graph: *lookup{=>_global}*
Identifier lookups, as done using the graph methods today, look up from a
cache representing the global environment.

Templates must not contribute to this environment until expansion.  Further,
metavariables will not be present in this environment.  To avoid confusion
and help obviate accidental contributions to this environment, the methods
have been renamed.  This will also allow for the creation of more general
methods down the line.

2023-03-29 12:58:35 -04:00
Mike Gerwitz 25121c1086 tamer: asg::air: Test formatting (token nesting)
This makes the tests quite a bit easier to understand visually.  I've been
doing this with all new tests but had to go back to some old ones, and still
have more to go back to.  Baby steps.

2023-03-29 12:58:35 -04:00
Mike Gerwitz bef68e1634 tamer: nir: Desugar shorthand template params and yield AIR
I had intended for this to be a full vertical slice initially, but AIR's
parser is going to need enough work that it'll muddy this patch a bit too

This keeps the desugaring simple, which is what I was hoping for.

The next step is to load it into the graph and emit regenerated longhand

I also don't like how the namespace prefix is just being ignored for
shorthand param desugaring.  This is also the case in the XSLT-based
compiler, but this violates TAMER's principle that it should parse every bit
of information; nothing should be ignored.  If something does not contribute
useful information, then it is not a useful construct and ought to be

2023-03-29 12:58:35 -04:00
Mike Gerwitz 3dcb2cb03c tamer: nir::NirEntity::TplParam: Optional name/value pair
This will be used for shorthand desugaring.

2023-03-29 12:58:35 -04:00
Mike Gerwitz a686855e9d tamer: Introduce desugaring operation for shorthand template application
This moves translation from NirToAir into TplShortDesugar, and changes the
output from AIR to NIR.

This is going to be much easier to reason about as a desugaring
operation (and indeed that's always how TAME has implemented it, in XSLT);
this keeps the complexity isolated.

Ideally, NirToAir wouldn't even accept tokens that it can't handle, but
that's going to take quite a bit more work and I don't have the time right
now.  Instead, we'll fail at runtime with some hopefully-useful
information.  It shouldn't actually happen in practice.

2023-03-29 12:58:34 -04:00
Mike Gerwitz 3e9f407527 tamer: asg::air::ir: Remove TplApply
The implementation decided upon in the previous commits have made this
unnecessary, using `RefIdent` to produce `Tpl->Ident[->Tpl]` instead.

2023-03-29 12:58:34 -04:00
Mike Gerwitz 669302700a tamer: build-aux/asg-ontviz: Vary arrowhead for cross edges
This makes it more visually apparent, when looking directly at a node,
whether an edge could represent a tree edge.

Dynamic edges could be tree edges, so I left those solid; that's the more
important visual indicator that I'm interested in, and it's disambiguated by
the dashed line.

2023-03-29 12:58:34 -04:00
Mike Gerwitz 893da0ed20 tamer: asg: Dynamically determined cross edges
Previous to this commit, ontological cross edges were declared
statically.  But this doesn't fare well with the decided implementation for
template application.

The documentation details it, but we have Tpl->Ident which could mean "I
define this Ident once expanded", or it could mean "this is a reference to a
template I will be applying".  The former is a tree edge, the latter is a
cross edge, and that determination can only be made by inspecting edge data
at runtime.

It could have been resolved by introducing new Object types, but that is a
lot of work for little benefit, especially given that only (right now) the
visitor uses this information.

2023-03-29 12:58:34 -04:00
Mike Gerwitz e132f108e8 tamer: asg::air: {=>diagnostic_}todo!
I forgot about my `diagnostic_todo!` macro!  The purpose was to help guide
development by obviating what comes next in test failures.

2023-03-29 12:58:34 -04:00
Mike Gerwitz 9d50157f8e tamer: Very basic support for template application NIR -> xmli
This this a big change that's difficult to break up, and I don't have the
energy after it.

This introduces nullary template application, short- and long-form.  Note
that a body of the short form is a `@values@` argument, so that's not
supported yet.

This continues to formalize the idea of what "template application" and
"template expansion" mean in TAMER.  It makes a separate `TplApply`
unnecessary, because now application is simply a reference to a
template.  Expansion and application are one and the same: when a template
expands, it'll re-bind metavariables to the parent context.  So in a
template context, this amounts to application.

But applying a closed template will have nothing to bind, and so is
equivalent to expansion.  And since `Meta` objects are not valid outside of
a `Tpl` context, applying a non-closed template outside of another template
will be invalid.

So we get all of this with a single primitive (getting the "value" of a

The expansion is conceptually like `,@` in Lisp, where we're splicing trees.

It's a mess in some spots, but I want to get this committed before I do a
little bit of cleanup.
2023-03-29 12:58:32 -04:00
Mike Gerwitz aa229b827c tamer: cargo clippy: Use active feature flags
This was missing `@FEATURES@`, which was causing more compilation than
necessary, but also causing clippy to evaluate different code.

This also adds RUSTFLAGS, for the same reason of not wanting to recompile.

2023-03-17 10:20:56 -04:00
Mike Gerwitz 03b46ebeff tamer: asg::air::tpl::TplState: Explicitly store reachability of active template
This is a small part of a larger change that I'm still working on.

2023-03-16 15:08:15 -04:00
Mike Gerwitz d930b26487 tamer: asg::air::ir: Decide on TplApply and expansion
This chooses Option B, as stated would likely be the case in the previous
commit.  The reasons are practical---I intend to support partial application
if doing so is worth it, either in implementation of the compiler or the
source language.

Closed templates can be referenced using `IdentRef` to trigger
expansion---their value is what they expand into, and they are spliced into
that point in the tree, like `,@` in Lisp.  We are able to overload this
behavior because we have the necessary type information.

However, I don't want to have to generate an Ident for every single template
expansion; there are many tens of thousands of them in our production
system.  Since AIR doesn't presently have a way to deal with this situation,
I'll for now add a special token that will close and expand a template in
place; it can be replaced with two separate tokens (`TplEnd` + `Ref`, for
example) in the future if such a need arises.

Are we there yet...?

2023-03-15 16:40:08 -04:00
Mike Gerwitz be81878dd7 tamer: src::asg: Scaffolding for metasyntactic variables
Also known as metavariables or template parameters.

This is a bit of a tortured excursion, trying to figure out how I want to
best represent this.  I have a number of pages of hand-written notes that
I'd like to distill over time, but the rendered graph ontology (via
`asg-ontviz`) demonstrates the broad idea.

`AirTpl::TplApply` highlights some remaining questions.  What I had _wanted_
to do is to separate the concepts of application and expansion, and support
partial application and such.  But it's going to be too much work for now,
when it isn't needed---partial application can be worked around by simply
creating new templates and duplicating params, as we do today, although that
sucks and is a maintenance issue.  But I'd rather address that head-on in
the future.

So it's looking like Option B is going to be the approach for now, with
templates being closed (as in, no free metavariables) and expanded at the
same time.  This simplifies the parser and error conditions significantly
and makes it easier to utilize anonymous templates, since it'll still be the
active context.

My intent is to get at least the graph construction sorted out---not the
actual expansion and binding yet---enough that I can use templates to
represent parts of NIR that do not have proper graph representations or
desugaring yet, so that I can spit them back out again in the `xmli` file
and incrementally handle them.  That was an option I had considered some
months ago, but didn't want to entertain it at the time because I wasn't
sure what doing so would look like; while it was an attractive approach
since it pushes existing primitives into the template system (something I've
wanted to do for years), I didn't want to potentially tank performance or
compromise the design for it after I had spent so much effort on all of this
so far.

But my efforts have yielded a system that significantly exceeds my initial
performance expectations, with a decent abstractions, and so this seems

2023-03-15 16:40:07 -04:00
Mike Gerwitz 9e5958d89e tamer: asg::air::ir::Air: Open/Close => Start/End in token names
See the Air docblock for more information.  I'm introducing new tokens for
the template system, which uses the terms "free" and "closed".  I prefer
open/close for delimiters, as I've expressed elsewhere, but unfortunately it
conflicts too much (and too confusingly) with other standard terminology as
we get more into the formal side of the language.

2023-03-15 10:59:25 -04:00
Mike Gerwitz 0e42788dcc tamer: asg::air: Restrict AirTplAggregate token domain to new AirTemplatable
This removes special cases, but it does complicate the parent `AirAggregate`
parser.  A pattern of delegation is forming, though abstracting it may be an
interesting challenge, given Rust's limitation on macro invocations as match
arms.  But, I think I can manage by generating the entire match using a
macro with a match-compatible syntax, augmenting where
needed...maybe.  This'll be messy.

...but if I can write the nightmare that is `ele_parse!`, I'm sure I can
manage this.  I just prefer to avoid complex macros unless I really need

2023-03-11 00:58:08 -05:00
Mike Gerwitz 2233c69bbf tamer: asg::graph::object: Some minor proofreading 2023-03-10 23:44:40 -05:00
Mike Gerwitz 18fa910e0f tamer: {tools=>build-aux}/asg-ontviz
Now that these are actually intended to be used as part of the build, this
is a more appropriate location.  I originally wrote it as a manual tool.

2023-03-10 15:13:30 -05:00
92 changed files with 12304 additions and 6082 deletions

View File

@ -20,7 +20,7 @@ build:
- export SAXON_CP=/usr/share/ant/lib/saxon9/saxon9he.jar
- export HOXSL=hoxsl
- ./bootstrap
- make all check info pdf html
- make clean all check info pdf html
- doc/

View File

@ -1,4 +1,4 @@
# This number is incremented for every linker change to force rebuilding
# of xmle files.
5 # Removal of {ret,}map:___{head,tail}

View File

@ -150,7 +150,7 @@
<xs:restriction base="xs:string">
<xs:pattern value="_[a-zA-Z0-9@\{\}-]+_|@[a-z][a-zA-Z0-9]*@" />
<xs:pattern value="_+[a-zA-Z0-9@\{\}-]+_+|@[a-z][a-zA-Z0-9]*@" />

View File

@ -127,27 +127,12 @@
$pkg-with-symtable/preproc:sym-deps" />
<!-- special fragment to be output as the head -->
<preproc:fragment id=":map:___head">
<!-- use a callback just in case we need to make portions of this async in the
future -->
<text>function( input, callback ) {</text>
<text>var output = {};</text>
<!-- compile mapped -->
<apply-templates select="./lvm:*" mode="lvmc:compile">
<with-param name="rater" select="$rater" />
<with-param name="type" select="'map'" />
<with-param name="symtable" select="$pkg-with-symtable/preproc:symtable"
<!-- special fragment to be output as the foot -->
<preproc:fragment id=":map:___tail">
<sequence select="$pkg-with-symtable/preproc:sym-deps/following-sibling::*" />
@ -204,27 +189,12 @@
$pkg-with-symtable/preproc:sym-deps" />
<!-- special fragment to be output as the head -->
<preproc:fragment id=":retmap:___head">
<!-- use a callback just in case we need to make portions of this async in the
future -->
<text>function( input, callback ) {</text>
<text>var output = {};</text>
<!-- compile mapped -->
<apply-templates select="./lvm:*" mode="lvmc:compile">
<with-param name="rater" select="$rater" />
<with-param name="type" select="'retmap'" />
<with-param name="symtable" select="$pkg-with-symtable/preproc:symtable"
<!-- special fragment to be output as the foot -->
<preproc:fragment id=":retmap:___tail">
<sequence select="$pkg-with-symtable/preproc:sym-deps/following-sibling::*" />
@ -236,19 +206,6 @@
<param name="type-prefix" select="'map'" />
<!-- purposely non-polluting. @ignore-dup is intended to be
temporary until static generation of these names is resolved;
this will not cause problems, since the code is always the
same (future bug pending!) -->
<preproc:sym name=":{$type-prefix}:___head"
no-deps="true" />
<preproc:sym name=":{$type-prefix}:___tail"
no-deps="true" />

View File

@ -657,10 +657,29 @@
select="lv:param-meta" />
<variable name="varname" select="@name" />
<variable name="param" select="$params[ @name=$varname ]" />
<!-- determine the nodes to copy -->
<variable name="copy"
select="$params[ @name=$varname ]/*" />
<variable name="copy" as="node()*">
<!-- TAMER desugared @values@ application convention (see -->
<when test="$param/@value">
<!-- the value is the name of a template to copy the body from -->
<variable name="dsgr" select="$param/@value" />
<!-- the template is always positioned as the immeditely-following
sibling of the `lv:apply-template` node -->
<sequence select="$params/parent::lv:apply-template
/following-sibling::lv:template[ @name=$dsgr ]
/*" />
<!-- old applicatication convention has child nodes within
`with-param` -->
<sequence select="$param/*" />
<!-- if the @expand flag is set, then immediately begin expanding any

View File

@ -148,7 +148,7 @@
<xs:restriction base="xs:string">
<xs:pattern value="_[a-zA-Z0-9@\{\}-]+_|@[a-z][a-zA-Z0-9]*@" />
<xs:pattern value="_+[a-zA-Z0-9@\{\}-]+_+|@[a-z][a-zA-Z0-9]*@" />

View File

@ -75,34 +75,16 @@
<variable name="retmap" select="/lv:package/l:retmap-exec" />
<!-- store a reference to the mapper in rater.fromMap() -->
<text>rater.fromMap = </text>
<when test="/lv:package/l:dep/preproc:sym[@type='map'][1]">
<value-of disable-output-escaping="yes" select="$map/text()" />
<!-- no map available -->
<!-- simply invoke the conintuation with the provided data -->
<text>; </text>
<text>var output={};</text>
<value-of disable-output-escaping="yes" select="$map/text()" />
<text>callback(output);}; </text>
<!-- return map -->
<text>rater._retmap = </text>
<when test="/lv:package/l:dep/preproc:sym[@type='retmap'][1]">
<value-of disable-output-escaping="yes" select="$retmap/text()" />
<!-- no map available -->
<!-- simply invoke the conintuation with the provided data -->
<text>; </text>
<text>var output={};</text>
<value-of disable-output-escaping="yes" select="$retmap/text()" />
<text>callback(output);}; </text>
<!-- we'll export a version that automatically performs the mapping -->
<text>module.exports = function( args_base, _canterm ) { </text>

tamer/Cargo.lock generated
View File

@ -116,6 +116,7 @@ dependencies = [

View File

@ -26,6 +26,7 @@ lto = true
arrayvec = ">= 0.7.1"
bumpalo = ">= 2.6.0"
exitcode = "1.1.2"
fixedbitset = ">= 0.4.1" # also used by petgraph
fxhash = ">= 0.2.1"
getopts = "0.2"
memchr = ">= 2.3.4" # quick-xml expects =2.3.4 at the time
@ -52,15 +53,9 @@ unicode-width = "0.1.5"
# This is enabled automatically for the `test` profile.
parser-trace-stderr = []
# Lowering NIR into AIR will begin adding objects to the graph. Since
# loweirng is not complete, this may result in errors. This flag can be
# removed when we are able to confidently state that this lowering will not
# cause issues in production TAME code.
wip-nir-to-air = []
# Derive `xmli` file from the ASG rather than a XIR token stream. This
# proves that enough information has been added to the graph for the entire
# program to be reconstructed. The `xmli` file will be a new program
# _derived from_ the original, and so will not match exactly.
wip-asg-derived-xmli = ["wip-nir-to-air"]
wip-asg-derived-xmli = []

View File

@ -52,7 +52,7 @@ rustdoc:
$(ontviz_svg): FORCE
tools/asg-ontviz | $(DOT) -Tsvg > $@
build-aux/asg-ontviz | $(DOT) -Tsvg > $@
# note that 'cargo check' is something else; see 'cargo --help'
test: check
@ -64,7 +64,7 @@ check-cargo:
.PHONY: check-docgen
tools/asg-ontviz >/dev/null
build-aux/asg-ontviz >/dev/null
.PHONY: check-system
check-system: bin
@ -72,7 +72,7 @@ check-system: bin
.PHONY: check-lint
.PHONY: check-fmt

View File

@ -1,471 +0,0 @@
// Abstract semantic graph benchmarks
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
// This file is part of TAME.
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <>.
// Note that the baseline tests have a _suffix_ rather than a prefix so that
// they are still grouped with the associated test in the output, since it's
// sorted lexically by function name.
extern crate tamer;
extern crate test;
use test::Bencher;
mod base {
use super::*;
use tamer::{
asg::{DefaultAsg, IdentKind, Source},
type Sut = DefaultAsg;
fn interned_n(n: u16) -> Vec<SPair> {
.map(|i| SPair(i.to_string().intern(), UNKNOWN_SPAN))
fn declare_1_000(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
bench.iter(|| {
.map(|i| sut.declare(*i, IdentKind::Meta, Source::default()))
fn declare_1_000_full_inital_capacity(bench: &mut Bencher) {
let mut sut = Sut::with_capacity(1024, 1024);
let xs = interned_n(1_000);
bench.iter(|| {
.map(|i| sut.declare(*i, IdentKind::Meta, Source::default()))
fn declare_1_000_prog_ident_size(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
bench.iter(|| {
.map(|i| sut.declare(*i, IdentKind::Meta, Source::default()))
fn declare_extern_1_000(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
bench.iter(|| {
.map(|i| {
sut.declare_extern(*i, IdentKind::Meta, Source::default())
fn resolve_extern_1_000(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
xs.iter().for_each(|sym| {
let _ =
sut.declare_extern(*sym, IdentKind::Meta, Source::default());
// Bench only the resolution, not initial declare.
bench.iter(|| {
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
// N.B.: This benchmark isn't easily comparable to the others because
// `set_fragment` takes ownership over a string, and so we have to clone
// strings for each call.
fn set_fragment_1_000_with_new_str(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
xs.iter().for_each(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
// Bench only the resolution, not initial declare.
bench.iter(|| {
.map(|sym| sut.set_fragment(*sym, "".into())) // see N.B.
fn lookup_1_000(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
xs.iter().for_each(|sym| {
let _ = sut.declare(*sym, IdentKind::Meta, Source::default());
bench.iter(|| {
.map(|sym| sut.lookup(*sym).unwrap())
fn get_1_000(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
bench.iter(|| {
.map(|oref| sut.get(*oref).unwrap())
// All dependencies on a single node. Petgraph does poorly with
// supernodes at the time of writing, relatively speaking.
fn add_dep_1_000_to_single_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
let root = orefs[0];
// Note that this adds all edges to one node
bench.iter(|| {
.map(|oref| sut.add_dep(root, *oref))
// Same as above but only one edge per node.
fn add_dep_1_000_one_edge_per_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
bench.iter(|| {
.map(|(from, to)| sut.add_dep(*from, *to))
fn has_dep_1_000_single_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
let root = orefs[0];
orefs.iter().for_each(|oref| {
sut.add_dep(root, *oref);
bench.iter(|| {
.map(|oref| sut.has_dep(root, *oref))
// Same as above but only one edge per node.
fn has_dep_1_000_one_edge_per_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
sut.declare(*sym, IdentKind::Meta, Source::default())
|(from, to)| {
sut.add_dep(*from, *to);
bench.iter(|| {
.map(|(from, to)| sut.has_dep(*from, *to))
fn add_dep_lookup_1_000_missing_one_edge_per_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
bench.iter(|| {
.map(|(from, to)| sut.add_dep_lookup(*from, *to))
fn add_dep_lookup_1_000_existing_one_edge_per_node(bench: &mut Bencher) {
let mut sut = Sut::new();
let xs = interned_n(1_000);
xs.iter().for_each(|sym| {
let _ = sut.declare(*sym, IdentKind::Meta, Source::default());
bench.iter(|| {
.map(|(from, to)| sut.add_dep_lookup(*from, *to))
mod object {
use super::*;
mod ident {
use super::*;
use tamer::{
asg::{Ident, IdentKind, Source},
span::UNKNOWN_SPAN as S0,
type Sut = Ident;
fn declare_1_000(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
(0..1000).map(|_| Sut::declare(sym)).for_each(drop);
fn resolve_1_000_missing(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
fn extern_1_000_missing(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
fn resolve_1_000_extern(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
.extern_(S0, IdentKind::Meta, Source::default())
.resolve(S0, IdentKind::Meta, Source::default())
fn resolve_1_000_override(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
Source {
virtual_: true,
Source {
override_: true,
// Override encountered before virtual
fn resolve_1_000_override_virt_after_override(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
Source {
override_: true,
Source {
virtual_: true,
fn set_fragment_1_000_resolved_with_new_str(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
.map(|_| {
.resolve(S0, IdentKind::Meta, Source::default())
// No need to do all of the others, since they're all the same thing.
fn declared_name_1_000(bench: &mut Bencher) {
let sym = SPair("sym".into(), S0);
bench.iter(|| {
(0..1000).map(|_| Sut::declare(sym).name()).for_each(drop);

View File

@ -1,109 +0,0 @@
// Benchmarking of sorting of ASG into xmle sections
// Copyright (C) 2014-2023 Ryan Specialty, LLC.
// This file is part of TAME.
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with this program. If not, see <>.
extern crate tamer;
extern crate test;
use test::Bencher;
pub(crate) use tamer::{
asg::{DefaultAsg, IdentKind, Source},
ld::xmle::{lower::sort, Sections},
type TestAsg = DefaultAsg;
fn interned_n(n: u16) -> Vec<SPair> {
.map(|i| SPair(i.to_string().intern(), UNKNOWN_SPAN))
fn sort_1_with_1_000_existing_supernode(bench: &mut Bencher) {
let mut sut = TestAsg::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
let root = orefs[0];
// All edges from a single node.
orefs.iter().skip(1).for_each(|to| {
sut.add_dep(root, *to);
bench.iter(|| {
drop(sort(&sut, Sections::new()));
fn sort_1_with_1_000_existing_one_edge_per_node_one_path(bench: &mut Bencher) {
let mut sut = TestAsg::new();
let xs = interned_n(1_000);
let orefs = xs
.map(|sym| {
// Note that there's no `cycle` call on the iterator, like the
// above tests, to make sure we don't create a cycle on the
// graph.