Commit Graph

1709 Commits (434365543e0319441f874f8cafd23c56d11418ba)

Author SHA1 Message Date
Mike Gerwitz 434365543e .gitlab-ci.yml: build: Clean before build
The motivating factor here is some out of date or corrupted rustc cache,
however we really ought to be doing fresh builds for TAME; it doesn't add
enough time that it's worth sacrificing assurances.
2023-05-19 13:43:41 -04:00
Mike Gerwitz e940fc5aa0 tamer: asg: Move index from Asg to AirAggregateCtx
This finally removes the awkward index from the ASG.  This will need much
more documentation and a better organized abstraction, but in the meantime,
previous commit dive into some of the rationale.

In essence: it only really makes sense to have indexing on the ASG itself if
it is used to cache queries or other expensive operations.  But that is not
what we were using it for---it was used for caching _lexical_ properties,
which are useful only during parsing for the sake of forming relationships
on the graph.  Once those relationships have formed, different types of
indexes will be useful in different lowering, optimization, or querying

This formalizes that, and in doing so, ensures that the index is will always
be accurate relative to the content of the ASG.  Once the index becomes
separated from it---through the `AirAggregateCtx::finish` operation---then
it is discarded and the ASG exposed.

This is also important because the index is incomplete---it contains only
the information necessary for the parser to carry out its task.

This change was a long time coming, and has reduced ASG to its essence.

2023-05-19 13:38:17 -04:00
Mike Gerwitz 7857460c1d tamer: Re-use prior AirAggreagteCtx for subsequent parsers
A new AirAggregate parser is utilized for each package import.  This
prevents us from moving the index from `Asg` onto `AirAggregateCtx` because
the index would be dropped between each import.

This allows re-using that context and solves for problems that result from
attempting to do so, as explained in the new
`resume_previous_parsing_context` test case.

But, it's now clear that there's a missing abstraction, and that reasoning
about this problem at the topmost level of the compiler/linker in terms of
internal parsing details like "context" is not appropriate.  What we're
doing is suspending parsing and resuming it later on for another package,
aggregating into the same destination (ASG + index).  An abstraction ought
to be formed in terms of that.

2023-05-19 13:38:15 -04:00
Mike Gerwitz 92214c7e05 tamer: asg::air: Include package in opaque identifier scope
This was the remaining of my stashed changes that I had mentioned in a
previous commit, but is accomplished differently than I had prototyped.  My
initial approach was a bit too klugey: to accept as an argument in various
scope contexts the active parser, as if it were the top stack frame.  This
was prototyped before the `AirPkgAggregate` parser was even created.

So we've since created a Pkg parser and now an opaque parser for opaque
idents.  There may be other opaque objects in the future.

Because of this change, the parent `AirPkgAggregate` gets stored on the
stack and just naturally becomes part of the lexical scope determination,
and so everything Just Works!

This commit was _supposed_ to be moving the index from `Asg` onto
`AirAggregateCtx`, but I wasn't able to do that because that context is
re-created for each package import currently.

2023-05-19 09:46:36 -04:00
Mike Gerwitz e266d42c48 tamer: asg::air::AirAggregateCtx: Tuple struct => named struct
As evidenced by this change, the tuple syntax was no longer serving us
well.  But the real reason for this change is to prepare for the addition of
a fourth field: the index, taken from `Asg`.

2023-05-18 01:32:28 -04:00
Mike Gerwitz 8b3dfe9149 tamer: asg::air: Utilize AirAggregateCtx for index lookups
This change means that `asg::air` is now the only module that directly
invokes index-related methods on `Asg`.  This clears the way, finally, to
removing the index from `Asg` entirely.

Not only does this result in a less awkward architecture, it also ensures
that lookups are forced to go through the system that understands and
controls lexical scoping, which will be able to give the correct answer.

Of course, the caveat is that the "correct" answer depends on what's
currently on the stack, depending on what type of lookup is being performed,
but those details are still encapsulated within the `asg::air` module and
its tests.

2023-05-18 01:10:23 -04:00
Mike Gerwitz 94bbc2d725 tamer: asg::air: Root AirIdent operations using AirAggregateCtx
This is the culmination of a great deal of work over the past few
weeks.  Indeed, this change has been prototyped a number of different ways
and has lived in a stash of mine, in one form or another, for a few weeks.

This is not done just yet---I have to finish moving the index out of Asg,
and then clean up a little bit more---but this is a significant
simplification of the system.  It was very difficult to reason about prior
approaches, and this finally moves toward doing something that I wasn't sure
if I'd be able to do successfully: formalize scope using AirAggregate's
stack and encapsulate indexing as something that is _supplemental_ to the
graph, rather than an integral component of it.

This _does not yet_ index the AirIdent operation on the package itself
because the active state is not part of the stack; that is one of the
remaining changes I still have stashed.  It will be needed shortly for
package imports.

This rationale will have to appear in docs, which I intend to write soon,
but: this means that `Asg` contains _resolved_ data and itself has no
concept of scope.  The state of the ASG immediately after parsing _can_ be
used to derive what the scope _must_ be (and indeed that's what
`asg::air::test::scope::derive_scopes_from_asg` does), but once we start
performing optimizations, that will no longer be true in all cases.

This means that lexical scope is a property of parsing, which, well, seems
kind of obvious from its name.  But the awkwardness was that, if we consider
scope to be purely a parse-time thing---used only to construct the
relationships on the graph and then be discarded---then how do we query for
information on the graph?  We'd have to walk the graph in search of an
identifier, which is slow.

But when do we need to do such a thing?  For tests, it doesn't matter if
it's a little bit slow, and the graphs aren't all that large.  And for
operations like template expansion and optimizations, if they need access to
a particular index, then we'll be sure to generate or provide the
appropriate one.  If we need a central database of identifiers for tooling
in the future, we'll create one then.  No general-purpose identifier lookup
_is_ actually needed.

And with that, `Asg::lookup_or_missing` is removed.  It has been around
since the beginning of the ASG, when the linker was just a prototype, so
it's the end of TAMER's early era as I was trying to discover exactly what I
wanted the ASG to represent.

2023-05-17 12:23:36 -04:00
Mike Gerwitz 716e217c9f tamer: asg: Restrict index-related operations to AIR
This is in the same spirit as previous commits modifying (or removing)
tests and benchmarks related to accessing the ASG and its indexes directly.

With this change, only `asg::air` uses the indexing and lookup methods on
`Asg`.  This will allow me to extract the index from `Asg` entirely and have
`Air` solely responsible for lookup; the graph will be responsible only for,
well, being a graph.  Indexing is an optimization strategy.

More information in the commit to follow.  But notice how this moving
environment-related concerns away from `Asg` and into AIR, and how the
remaining environment concerns are index-related.

But there is one remaining barrier: to fully move the indexing away from
`Asg`, we have to use an alternative (and complete)
abstraction---AirAggregateCtx with its ability to resolve and introduce
scope based on the stack.  The `AirIdent` token subset doesn't yet do that,
and all the work up to this point was in prepartion for doing that.  Since
introducing indexing at Root a few commits ago, it's now possible to

2023-05-17 11:37:03 -04:00
Mike Gerwitz 92eb991df3 tamer: benches: Remove asg and asg_lower_xmle microbenchmarks
These benchmarks were useful as TAMER was in its infancy and I was trying to
gain an intuition for working with Rust.  But they are now out of date, and
there are better ways to measure TAMER's performance, including running it
on real-world data (which wasn't possible previously) and through profiling
tools like Valgrind.

With that said, these types of benchmarks _would_ be useful for helping to
dig down into improvements that could be made, at a glance.  The problem is,
they aren't testing anything new, and they're also testing something I'm
about to extract from `Asg`.  It is not worth the ongoing maintenance cost.

So benchmarks may be reintroduced in the future if they are found to be

2023-05-17 11:14:00 -04:00
Mike Gerwitz b61e1ce952 tamer: asg::air: Common asg_from_toks for tests
The previous commit introduced a duplicate `asg_from_toks`; this just makes
it available publicly for any tests that might utilize AIR to lower the
barrier to writing such tests and provide some guidance in doing so.

2023-05-17 10:57:10 -04:00
Mike Gerwitz 79fa10f26b tamer: ld::xmle::lower::test: Use AIR (decouple from Asg and index)
This uses AIR---the ASG's proper public interface now---to construct the
graph for tests, just as all the other modern tests do.  This is change
works towards encapsulating index operations (both creation and lookups) so
that the index can be moved off of Asg and into AIR, where it belongs.  More
information on that and rationale to come.

2023-05-17 10:50:57 -04:00
Mike Gerwitz b238366bee tamer: asg::air::test::scope: Individual test per scope assertion
This separates the previous scope assertions per identifier into individual
tests, so that a single failure will not preempt all others.

2023-05-17 00:25:31 -04:00
Mike Gerwitz bfbaa6e528 tamer: asg::air::test::scope: More useful panic function/line
This will panic within the test function so that RUST_BACKTRACE is not

It also prepares for the changes coming up next.

2023-05-16 23:49:23 -04:00
Mike Gerwitz ba38a3c1ba tamer: src::asg::air: Pool identifiers into global environment
This, finally, introduces identifier pooling in the global environment,
represented by `Root`.  All package-level identifiers will be scoped as
such, which at the moment means anything that's not within a template.

As mentioned in recent commits, this does require additional cleanup to
finalize, and some more test will make additional rationale more clear.

It's also worth noting the intent of storing the `ObjectIndex<Root>`---not
only does it mean that the active root can be derived solely from the
current parsing state, but it also means that in the future we can
contribute to any, potentially multiple, roots.  I had previously used Neo4J
to effectively diff two dependency graphs between versions in the current
XSLT-based TAMER; I'd like to be able to do that with TAMER in the future,
which is an important concept when considering automated data migration, as
well as querying for the effects of changes.

More to come.  I'm hoping this is finally nearing a conclusion and I can
finally tie everything together with package imports.  `AirIdent` will be
introduced into the mix soon now too, now that this commit is able to root

2023-05-16 23:28:47 -04:00
Mike Gerwitz 1cf5488756 tamer: asg::air::EnvScopeKind::Pool: Remove (Visible is sufficient)
At least, I think so.

See previous commit for more information, and the commit that follows for
actually using it at Root.

2023-05-16 23:28:45 -04:00
Mike Gerwitz 33f34bf244 tamer: asg: Initial identifier scoping
Okay, this is finally distilling into something fairly simple and
reasonable, but I'm not quite there yet.

In particular, the responsibility is simply between `Asg` (as the owner of
the index) and `AirAggregateCtx` (as the owner of the stack frames from
which environments and scope are derived).  This was inevitable and I was
waiting for it, but now I have a good idea of how to clean it up and

This also doesn't index in root yet (`active_rooting_oi` is still `None` for
`Root`), and I think I may remove `Pool` and just make it `Visible` at that
point, since it won't be going any further anyway.  I don't think the
distinction is meaningful and will just complicate implementations.

The tests also need some more cleanup---the assertions ideally would live in
independent tests, and the assertion failure is in a function call rather
than the test (function) itself, so requires a Rust backtrace to locate the
line number of (unless you look at the failure data).

So I suppose this is more of a mental synchronization point than
anything.  Nothing's broken, though.

2023-05-16 14:58:21 -04:00
Mike Gerwitz 9fb2169a06 tamer: asg::air: Begin to introduce explicit scope testing
There's a lot of documentation on this in the commit itself, but this stems

  a) frustration with trying to understand how the system needs to operate
     with all of the objects involved; and
  b) recognizing that if I'm having difficulty, then others reading the
     system later on (including myself) and possibly looking to improve upon
     it are going to have a whole lot of trouble.

Identifier scope is something I've been mulling over for years, and more
formally for the past couple of months.  This finally begins to formalize
that, out of frustration with package imports.  But it will be a weight
lifted off of me as well, with issues of scope always looming.

This demonstrates a declarative means of testing for scope by scanning the
entire graph in tests to determine where an identifier has been
scoped.  Since no such scoping has been implemented yet, the tests
demonstrate how they will look, but otherwise just test for current
behavior.  There is more existing behavior to check, and further there will
be _references_ to check, as they'll also leave a trail of scope indexing
behind as part of the resolution process.

See the documentation introduced by this commit for more information on
that part of this commit.

Introducing the graph scanning, with the ASG's static assurances, required
more lowering of dynamic types into the static types required by the
API.  This was itself a confusing challenge that, while not all that bad in
retrospect, was something that I initially had some trouble with.  The
documentation includes clarifying remarks that hopefully make it all

2023-05-12 14:07:29 -04:00
Mike Gerwitz 00ff660008 tamer: asg::air: Begin lexical identifier resolution from bottom up
This begins demonstrating that the root will be utilized for identifier
lookup and indexing, as it was originally for TAME and is currently for the

This was _not_ the original plan---the plan was to have identifiers indexed
only at the package level, at least until we need a global lookup for
something else---but that plan was upended by how externs are currently
handled.  So, for now, we need a global scope.

(Externs are resolved by the linker in such a way that _any_ package that
happens to be imported transitively may resolve the import.  This is a
global environment, which I had hoped to get rid of, and which will need to
eventually go away (possibly along with externs) to support loading multiple
programs into the graph simultaneously for cross-program analysis.)

This commit renames the base state for `AirAggregate` to emphasize the fact,
especially when observing it in the `AirStack`, and changes
`AirAggregateCtx::lookup_lexical_or_missing` to resolve from the _bottom_ of
the stack upward, rather than reverse, to prove that the system still
operates correctly with this change in place.

The reason for this direction change is to simplify lookup in the most
general case of non-local identifiers, which are almost all of them in
practice---they'll be immediately resolved at the root once they're
indexed.  This can be done because I determined that I will _not_ support
shadowing; rationale for that will come later, but TAME is intended to be a
language suitable for non-programmer audiences as well.  Note that
identifiers will be resolved lexically within templates in TAMER, unlike
TAME, which means that the expansion context will _not_ be considered when
checking for shadowing, so templates will still be able to compose without a
problem so long as they do not shadow in their definition context.  (I'll
have to consider how that affects template-generating templates later on,
but that's an ambiguous construction in TAME today anyway.)

This _does not_ yet index anything at the root where it wasn't already being
indexed explicitly.

2023-05-10 14:43:33 -04:00
Mike Gerwitz dd6a6dd196 tamer: asg::air::ir::AirPkg::PkgStart: Require name
This requires the name as part of the package definition, which in turn
removes a state (and all the combinations resulting from it) from
AirAggregate, which results in significant complexity reduction for a very
complex part of the system.

Pushing this complexity outward results in a reduction of overall
complexity, and obviates the question of where NIR will receive a generated

2023-05-10 13:57:45 -04:00
Mike Gerwitz 7a6aef00b2 tamer: nir::air::NirToAir: Note about intent to refactor
The comment speaks for itself.

My concern is that this will be especially off-putting to people looking at
TAMER and wondering how one could possibly work with this system.

2023-05-10 13:57:41 -04:00
Mike Gerwitz 4510de38ed tamer: asg::air::pkg: Extract AirPkgAggregate from AirAggregate
This is something I've wanted to do for some time, but the system is
becoming hard enough to reason about (with some attempted future changes)
that I require the consistency afforded by this change.

It's not entirely done---as noted by the TODO for `UnnamedPkg`---but it's
close, and then `AirAggregate` will just be a delegating superstate, like

Importantly, this also puts a package parser on the stack, which will work
better with the stack-based scoping system being developed.  It will also
make it easier to fall back to a base case that I had really wanted to
avoid, and will have more information on in the future: root indexing for a
shared global environment for package-level identifiers.  (Imports are still
package-scoped, but only in appearance, by contributing to the global
environment of the compilation unit during import.  Well, it doesn't do that
yet.  The XSLT compiler works in that way.)

2023-05-10 13:57:39 -04:00
Mike Gerwitz ebdae9ac38 tamer: ld::xmle::lower: Sort only rooted Idents
This is one of many changes that have been lingering that I need to start to
break apart in an attempt to commit the confusing and disappointing
conclusion to this package loading madness.

More information to come.

2023-05-09 15:20:39 -04:00
Mike Gerwitz 5f275fb801 tamer: asg::air::air: Eliminate todo! for unexpected AirIdent
I had apparently forgotten about this, because I didn't benefit from the
exhaustiveness check; this needs to be eliminated so that this doesn't
happen again, and to provide a proper non-panicking error.

2023-05-09 12:35:06 -04:00
Mike Gerwitz 4ec4857360 Revert "tamer: asg::air::ir::AirBind::RefIdent: New optional canonical name"
This reverts commit da7fe96254e425bc7b75f8cf454465b71e27e372.

I'm a fool---this would be pursuant to a future plan that removes AirIdent
opaque tokens.  But for now, I need it on IdentDecl and others, which
currently has a `Source` (that I want to go away, as just mentioned), which
contains the same information.

So maybe more to come on this...

2023-05-09 12:35:06 -04:00
Mike Gerwitz 572337505c tamer: asg::air::ir::AirBind::RefIdent: New optional canonical name
This allows for a canonical package name to be optionally provided to
explicitly resolve a reference against, avoiding a lexical lookup.

This change doesn't actually utilize this new value yet; it just
retains BC.  The new argument will be used for the linker, since it already
knows the package that defined an identifier while reading the object file's
symbol table.  It will also be used by tamec for the same purposes while
processing package imports.


-- squashed with --

tamer: asg::air::ir::RefIdent: CanonicalName=SPair

The use of CanonicalName created an asymmetry between RefIdent and
BindIdent.  The hope was to move CanonicalName instantiation outside of AIR
and into NIR, but doing so would be confusing and awkward without doing
something with BindIdent.

I don't have the time to deal with that for now, so let's observe how the
system continues to evolve and see whether hoisting it out makes sense in the
end.  For now, this works just fine and I need to move on with the actual
goal of finishing package imports so that I can expand templates.

2023-05-09 12:35:06 -04:00
Mike Gerwitz 00492ace01 tamer: obj::xmlo: Bind packages to canonical name
NOTE: This fixes the aforementioned commit that caused the linker to
temporarily fail (670c5d3a5d at time of
writing).  This does introduce an extra forward slash into
`l:dep/preproc:sym/@src`, but that does not appear to cause any
problems.  That will eventually go away, so I'm not going to bother with it
any further.

As the `xmlo` file is lowered into AIR, the name will be prefixed with a
leading slash (if necessary, which it is atm) and will emit an

This means that packages will be properly indexed by their canonical name on
load, which will be important when we share this with tamec.

2023-05-09 12:34:12 -04:00
Mike Gerwitz 13bac8382f tamer: obj::xmlo::{air,reader}::test: Format test cases
Simple reformatting that's consistent with other more recent tests, before I
go making changes.

2023-05-05 10:26:58 -04:00
Mike Gerwitz 48bcb0cdab tamer: asg: Integrate package CanonicalName
This change requires every package to have a canonical name, and performs
namespec canonicalization on imports.

Since all package names are canonicalized, this opens the door to being able
to index package names at import, allowing the object to be shared on the
graph and properly reference a package after it has been resolved.

Note that the system tests' canonicalization is relative to the hard-coded
`/TODO` presently; that will change in the near future once `tamec`
generates names from the provided path.

2023-05-05 10:26:58 -04:00
Mike Gerwitz a9d0f43684 tamer: src::asg::graph::object::pkg::name: New module
This introduces, but does not yet integrate, `CanonicalName`, which not only
represents canonicalized package names, but handles namespec resolution.

The term "namespec" is motivated by Git's use of *spec (e.g. refspec)
referring to various ways of specifying a particular object.  Names look
like paths, and are derived from them, but they _are not paths_.  Their
resolution is a purely lexical operation, and they include a number of
restrictions to simplify their clarity and handling.  I expect them to
evolve more in the future, and I've had ideas to do so for quite some time.

In particular, resolving packages in this way and then loading the from the
filesystem relative to the project root will ensure that
traversing (conceptually) to a parent directory will not operate
unintuitively with symlinks.  The path will always resolve unambigiously.

(With that said, if the symlink is to a shared directory with different
directory structures, that doesn't solve the compilation problem---we'll
have to move object files into a project-specific build directory to handle

Span Slicing
Okay, it's worth commenting on the horridity of the path name slicing that
goes on here.  Care has been taken to ensure that spans will be able to be
properly sliced in all relevant contexts, and there are plenty of words
devoted to that in the documentation committed here.

But there is a more fundamental problem here that I regret not having solved
earlier, because I don't have the time for it right now: while we do have
SPair, it makes no guarantees that the span associated with the corresponding
SymbolId is actually the span that matches the original source lexeme.  In
fact, it's often not.

This is a problem when we want to slice up a symbol in an SPair and produce
a sensible span.  If it _is_ a source lexeme with its original span, that's
no problem.  But if it's _not_, then the two are not in sync, and slicing up
the span won't produce something that actually makes sense to the user.  Or,
worse (or maybe it's not worse?), it may cause a panic if the slicing is out
of bounds.

The solution in the future might be to store explicitly the state of an
SPair, or call it Lexeme, or something, so that we know the conditions under
which slicing is safe.  If I ever have time for that in this project.

But the result of the lack of a proper abstraction really shows here: this
is some of the most confusing code in TAMER, and it's really not doing
anything all that complicated.  It is disproportionately confusing.

2023-05-05 10:26:56 -04:00
Mike Gerwitz e3a68aaf9e tamer: span: Introduce rslice, slice_{head,tail}
These are used by an upcoming commit, where I'll have much more to say about
the topic of span slicing.

2023-05-05 10:25:35 -04:00
Mike Gerwitz 670c5d3a5d tamer: asg::graph: Require name for non-imports
NOTE: This temporarily breaks `tameld`.  It is fixed in a future commit when
names are bound.  This was an oversight when breaking apart changes into
separate commits, because the linker does not yet have system tests like
tamec does.

This is preparing for a full transition to requiring a canonical package
name.  The previous `Unnamed` variant has been removed and `AirAggregate`
will provide a default `WS_EMPTY` name, as `Pkg` had done before.

The intent of this change is to allow for consulting the index before a
new `Pkg` object is created on the graph, but we're not quite ready for that

Well, that's not entirely true---the linker can be ready for that.  But the
compiler needs to canonicalize import paths relative to the active package
canonical name, which it can't even do yet because tamec isn't generating a

So maybe the linker will be first; it's useful to have that in a separate
commit anyway to emphasize the change.

2023-05-05 10:24:47 -04:00
Mike Gerwitz 799f2c6d96 tamer: tameld: Produce first error
...this has apparently been consuming errors for some time.  This would
cause the parser to enter an invalid state in some cases and terminate.

This would _not_ permit an invalid link, as the graph would not be correct,
but it was masking the actual error.

This part of linker is in dire need of tests.  This also ought to be
replaced with tamec's approach of reporting all errors.

2023-05-04 16:04:52 -04:00
Mike Gerwitz 7cfe6a6f8d tamer: asg::graph: Index Root->Pkg with canonical names
The previous commit introduced canonical names, and this uses them to index.

The next step will be to utilize those names to look up packages on
definition rather than creating a new package node, so that references to
yet-to-be-defined (or yet-to-be-imported) packages can be resolved on the

2023-05-02 16:15:07 -04:00
Mike Gerwitz 92c9c9ba2f tamer: asg: Introduce package canonical name concept
This is already a concept in the XSLT-based compiler, where each package has
a `package/@name` generated from its path.  The same will happen with tamec.

Before we can load packages into the graph, we need canonical identifiers so
that they can be indexed.  The next commit will handle indexing using this

2023-05-02 16:08:39 -04:00
Mike Gerwitz 56ab671363 tamer: nir::interp: Ignore Text tokens
The documentation explains the intent here---existing LaTeX documentation.

The intent was to simply copy the documentation into a LaTeX document based
on the lvspec package that I had created long ago.  Of course, that's not
appropriate---we're a DSL and should provide first-class support for
documentation that will compile properly into the target format, whether it
be LaTeX, HTML, JS, or anything else.

2023-04-30 15:06:58 -04:00
Mike Gerwitz 068804b397 tamer: Remove {ret}map:___{head,tail}
These have been a pain in the ass since TAMER began.

It seemed like a good idea at the time to have static code generated in this
way, but the lack of explicit dependencies just makes this a mess and works
against the operating theory of the system.

Furthermore, the _same_ static fragments were generated for each and every
map package.

There is still a post-link step (standalones) handled in XSLT; the
previously-static code has been moved there.  This will eventually be
integrated into tameld itself, once TAMER has facilities for JS generation.

(This was discovered while trying to parent identifiers to packages.)

2023-04-30 15:06:47 -04:00
Mike Gerwitz 77ada079e1 tamer: asg::graph::Asg.graph: Finally encapsulate
With the previous commit using a visitor implemented within the `asg`
module, we can now finally encapsulate the graph.  This is a wonderfully
liberating, long-awaited change, since I have been fighting with the lack of
encapsulation for some time; it has made certain changes challenging and has
made the system more difficult to reason about.  It also made it impossible
to assert that invariants were _actually_ properly enforced, if things could
just peer into and modify the graph directly, out from underneath the API
that provides those assurances.

This also removes our dependency on Petgraph outside of the `asg`
module.  There are no plans to migrate away from it currently; we'll see how
the graph continues to evolve over time and what redundancies are introduced
with our data structures.  It may render petgraph unnecessary.

Interestingly, because my DFS implementation is so similar to Petgraph's,
the emitted ordering is _identical_ between this commit and the previous.

2023-04-28 15:36:07 -04:00
Mike Gerwitz 78c1a9136e tamer: ld::xmle::lower: Use asg::graph::visit::topo::topo_sort
This integrates the new topological sort, replacing the previous
implementation in the linker.

This will now allow encapsulating the graph, finally, and ensures that
future changes can be fully maintained within the `asg` module.

More cleanup will come over time.

2023-04-28 15:26:47 -04:00
Mike Gerwitz 9b53a5e176 tamer: asg::graph::visit::topo: Cut cycles
This commit includes plenty of documentation, so you should look there.

It's desirable to describe the sorting that TAME performs as a topological
sort, since that's the end result we want.  This uses the ontology to
determine what to do to the graph when a cycle is encountered.  So
technically we're sorting a graph with cycles, but you can equivalently view
this as first transforming the graph to cut all cycles and then sorting it.

For the sake of trivia, the term "cut" is used for two reasons: (1) it's an
intuitive visualization, and (2) the term "cut" has precedence in logic
programming (e.g. Prolog), where it (`!`) is used to prevent
backtracking.  We're also preventing backtracking, via a back edge, which
would produce a cycle.

2023-04-28 14:33:48 -04:00
Mike Gerwitz c2c1434afe tamer: asg::graph::visit::topo: Cycle detection
This introduces cycle detection, but it does not yet filter ontologically
permitted cycles, which will be needed prior to utilizing this in `tameld`.

There's a considerable amount of documentation here.  While the
implementation is fairly simple, there are important algorithmic decisions,
both in the DFS construction and the derivation of the cycle path from data
that already exists.

This also supports recovery (by ignoring cycles), which can then be utilized
to find more cycles and other errors in the system.

2023-04-27 16:28:57 -04:00
Mike Gerwitz e3094e0bad tamer: asg::graph::visit::topo: Introduce topological sort
This is an initial implementation that does not yet produce errors on
cycles.  Documentation is not yet complete.

The implementation is fairly basic, and similar to Petgraph's DFS.

A terminology note: the DFS will be ontology-aware (or at least aware of
edge metadata) to avoid traversing edges that would introduce cycles in
situations where they are permitted, which effectively performs a
topological sort on an implicitly _filtered_ graph.

This will end up replacing ld::xmle::lower::sort.

2023-04-26 09:51:45 -04:00
Mike Gerwitz be05fbb833 tamer: asg::graph::visit{=>::ontree}: Move into submodule
This reorganization makes way for more traversals.

2023-04-24 13:51:04 -04:00
Mike Gerwitz 42aa5bd407 tamer: asg::graph: Root->Ident {tree=>cross} edge
tameld isn't yet adding edges to Idents from their associated Pkg (see
previous commit), but this formalizes how the ontology will interpret such a
relationship.  The idea is that Idents are always owned by Pkgs, but they
may be optionally explicitly rooted, which will be used by a particular type
of DFS walk that is about to be written, which can ignore Root->Pkg and
focus instead on cross edges to Idents.

Though it's not lost on me that now that I'll be introducing a DFS for the
linker, the terms "cross" and "tree" edge now become ambiguous; I used to
call them "ontological X edge", but I had fallen out of that habit; perhaps
I need to reintroduce that rigor.

2023-04-24 09:44:02 -04:00
Mike Gerwitz 48d9bca3b7 tamer: obj::xmlo: Add Pkg nodes for identifiers
This modifies the xmlo reader, xmlo->AIR lowering, and AIR->ASG to introduce
a package for identifiers.  It does not yet, however, add edges from the
package to the identifier.

Once edges are added, the DFS will change in undesirable ways, which will
require a new implementation.  This is desirable to decouple from Petgraph
anyway, and then will be able to restore the prior single-pass sort+cycle

That will also encapsulate visiting behavior within the `asg::graph` module
and, in turn, allow encapsulating `Asg.graph` finally.

2023-04-21 16:24:11 -04:00
Mike Gerwitz 34dad122fd tamer: asg::air::test: Remove `Air` and `Parsed` enum prefixes from variants
Same spirit as previous commits.  This is committed separately from the
changes that follow to eliminate its distraction.

2023-04-21 15:37:27 -04:00
Mike Gerwitz 1f371b6ba6 tamer: obj::xmlo::air: More explicit dead states
This doesn't go far enough, but it elaborates a bit---the existing was far
too much of a catch-all.  It's important to take advantage of exhaustiveness
checks to ensure each transition is properly accounted for.

This parser is going to get more work over time, including right now, so I'm
not going to go too deep into this yet, but it's be useful (as a reader) to
compare it to e.g. asg::air's parsers' explicit enumeration of states and
favoring of explicit errors over dead state transitions.

2023-04-21 11:48:56 -04:00
Mike Gerwitz dd47fc564d tamer: obj::xmlo: More concise identifiers
This follows conventions of other, more recently written, systems.

2023-04-21 11:48:55 -04:00
Mike Gerwitz e13817c203 tamer: obj::xmlo::air::test: Extract into own file 2023-04-20 16:46:32 -04:00
Mike Gerwitz 6f68292df5 tamer: asg::graph::{index_identifier=>index}: Generalize
This may now index _any_ type of object, in preparation for indexing package
import paths.  In practice, this only makes sense (at least currently) for
`Pkg` and `Ident`.

This generalization also applies to `Asg::lookup_or_missing`.

2023-04-20 16:46:30 -04:00
Mike Gerwitz f183600c3a tamer: asg: Move Ident-specific methods off of Asg
Historically, the ASG was better described as a "dependency graph",
containing only identifiers (which are simply called "symbols" in the
XSLT-based compiler).  Consequently, it was appropriate for the graph to
have operations specific to identifiers.  (Indeed, that's the only type of
object the graph supported.)

Much has changed since then.  This cleans things up, and makes parenting
identifiers to root an _explicit_ operation.  This will make it easier to
move forward with handling of scope, and importing identifiers into
packages, and removing `Source`, and so on.

2023-04-19 12:40:35 -04:00