NOTE: This fixes the aforementioned commit that caused the linker to
temporarily fail (670c5d3a5d at time of
writing). This does introduce an extra forward slash into
`l:dep/preproc:sym/@src`, but that does not appear to cause any
problems. That will eventually go away, so I'm not going to bother with it
any further.
As the `xmlo` file is lowered into AIR, the name will be prefixed with a
leading slash (if necessary, which it is atm) and will emit an
`Air::BindIdent`.
This means that packages will be properly indexed by their canonical name on
load, which will be important when we share this with tamec.
DEV-13162
This change requires every package to have a canonical name, and performs
namespec canonicalization on imports.
Since all package names are canonicalized, this opens the door to being able
to index package names at import, allowing the object to be shared on the
graph and properly reference a package after it has been resolved.
Note that the system tests' canonicalization is relative to the hard-coded
`/TODO` presently; that will change in the near future once `tamec`
generates names from the provided path.
DEV-13162
This introduces, but does not yet integrate, `CanonicalName`, which not only
represents canonicalized package names, but handles namespec resolution.
The term "namespec" is motivated by Git's use of *spec (e.g. refspec)
referring to various ways of specifying a particular object. Names look
like paths, and are derived from them, but they _are not paths_. Their
resolution is a purely lexical operation, and they include a number of
restrictions to simplify their clarity and handling. I expect them to
evolve more in the future, and I've had ideas to do so for quite some time.
In particular, resolving packages in this way and then loading the from the
filesystem relative to the project root will ensure that
traversing (conceptually) to a parent directory will not operate
unintuitively with symlinks. The path will always resolve unambigiously.
(With that said, if the symlink is to a shared directory with different
directory structures, that doesn't solve the compilation problem---we'll
have to move object files into a project-specific build directory to handle
that.)
Span Slicing
------------
Okay, it's worth commenting on the horridity of the path name slicing that
goes on here. Care has been taken to ensure that spans will be able to be
properly sliced in all relevant contexts, and there are plenty of words
devoted to that in the documentation committed here.
But there is a more fundamental problem here that I regret not having solved
earlier, because I don't have the time for it right now: while we do have
SPair, it makes no guarantees that the span associated with the corresponding
SymbolId is actually the span that matches the original source lexeme. In
fact, it's often not.
This is a problem when we want to slice up a symbol in an SPair and produce
a sensible span. If it _is_ a source lexeme with its original span, that's
no problem. But if it's _not_, then the two are not in sync, and slicing up
the span won't produce something that actually makes sense to the user. Or,
worse (or maybe it's not worse?), it may cause a panic if the slicing is out
of bounds.
The solution in the future might be to store explicitly the state of an
SPair, or call it Lexeme, or something, so that we know the conditions under
which slicing is safe. If I ever have time for that in this project.
But the result of the lack of a proper abstraction really shows here: this
is some of the most confusing code in TAMER, and it's really not doing
anything all that complicated. It is disproportionately confusing.
DEV-13162
NOTE: This temporarily breaks `tameld`. It is fixed in a future commit when
names are bound. This was an oversight when breaking apart changes into
separate commits, because the linker does not yet have system tests like
tamec does.
This is preparing for a full transition to requiring a canonical package
name. The previous `Unnamed` variant has been removed and `AirAggregate`
will provide a default `WS_EMPTY` name, as `Pkg` had done before.
The intent of this change is to allow for consulting the index before a
new `Pkg` object is created on the graph, but we're not quite ready for that
yet.
Well, that's not entirely true---the linker can be ready for that. But the
compiler needs to canonicalize import paths relative to the active package
canonical name, which it can't even do yet because tamec isn't generating a
name.
So maybe the linker will be first; it's useful to have that in a separate
commit anyway to emphasize the change.
DEV-13162
...this has apparently been consuming errors for some time. This would
cause the parser to enter an invalid state in some cases and terminate.
This would _not_ permit an invalid link, as the graph would not be correct,
but it was masking the actual error.
This part of linker is in dire need of tests. This also ought to be
replaced with tamec's approach of reporting all errors.
DEV-13162
The previous commit introduced canonical names, and this uses them to index.
The next step will be to utilize those names to look up packages on
definition rather than creating a new package node, so that references to
yet-to-be-defined (or yet-to-be-imported) packages can be resolved on the
graph.
DEV-13162
This is already a concept in the XSLT-based compiler, where each package has
a `package/@name` generated from its path. The same will happen with tamec.
Before we can load packages into the graph, we need canonical identifiers so
that they can be indexed. The next commit will handle indexing using this
information.
DEV-13162
The documentation explains the intent here---existing LaTeX documentation.
The intent was to simply copy the documentation into a LaTeX document based
on the lvspec package that I had created long ago. Of course, that's not
appropriate---we're a DSL and should provide first-class support for
documentation that will compile properly into the target format, whether it
be LaTeX, HTML, JS, or anything else.
DEV-13162
These have been a pain in the ass since TAMER began.
It seemed like a good idea at the time to have static code generated in this
way, but the lack of explicit dependencies just makes this a mess and works
against the operating theory of the system.
Furthermore, the _same_ static fragments were generated for each and every
map package.
There is still a post-link step (standalones) handled in XSLT; the
previously-static code has been moved there. This will eventually be
integrated into tameld itself, once TAMER has facilities for JS generation.
(This was discovered while trying to parent identifiers to packages.)
DEV-13162
With the previous commit using a visitor implemented within the `asg`
module, we can now finally encapsulate the graph. This is a wonderfully
liberating, long-awaited change, since I have been fighting with the lack of
encapsulation for some time; it has made certain changes challenging and has
made the system more difficult to reason about. It also made it impossible
to assert that invariants were _actually_ properly enforced, if things could
just peer into and modify the graph directly, out from underneath the API
that provides those assurances.
This also removes our dependency on Petgraph outside of the `asg`
module. There are no plans to migrate away from it currently; we'll see how
the graph continues to evolve over time and what redundancies are introduced
with our data structures. It may render petgraph unnecessary.
Interestingly, because my DFS implementation is so similar to Petgraph's,
the emitted ordering is _identical_ between this commit and the previous.
DEV-13162
This integrates the new topological sort, replacing the previous
implementation in the linker.
This will now allow encapsulating the graph, finally, and ensures that
future changes can be fully maintained within the `asg` module.
More cleanup will come over time.
DEV-13162
This commit includes plenty of documentation, so you should look there.
It's desirable to describe the sorting that TAME performs as a topological
sort, since that's the end result we want. This uses the ontology to
determine what to do to the graph when a cycle is encountered. So
technically we're sorting a graph with cycles, but you can equivalently view
this as first transforming the graph to cut all cycles and then sorting it.
For the sake of trivia, the term "cut" is used for two reasons: (1) it's an
intuitive visualization, and (2) the term "cut" has precedence in logic
programming (e.g. Prolog), where it (`!`) is used to prevent
backtracking. We're also preventing backtracking, via a back edge, which
would produce a cycle.
DEV-13162
This introduces cycle detection, but it does not yet filter ontologically
permitted cycles, which will be needed prior to utilizing this in `tameld`.
There's a considerable amount of documentation here. While the
implementation is fairly simple, there are important algorithmic decisions,
both in the DFS construction and the derivation of the cycle path from data
that already exists.
This also supports recovery (by ignoring cycles), which can then be utilized
to find more cycles and other errors in the system.
DEV-13162
This is an initial implementation that does not yet produce errors on
cycles. Documentation is not yet complete.
The implementation is fairly basic, and similar to Petgraph's DFS.
A terminology note: the DFS will be ontology-aware (or at least aware of
edge metadata) to avoid traversing edges that would introduce cycles in
situations where they are permitted, which effectively performs a
topological sort on an implicitly _filtered_ graph.
This will end up replacing ld::xmle::lower::sort.
DEV-13162
tameld isn't yet adding edges to Idents from their associated Pkg (see
previous commit), but this formalizes how the ontology will interpret such a
relationship. The idea is that Idents are always owned by Pkgs, but they
may be optionally explicitly rooted, which will be used by a particular type
of DFS walk that is about to be written, which can ignore Root->Pkg and
focus instead on cross edges to Idents.
Though it's not lost on me that now that I'll be introducing a DFS for the
linker, the terms "cross" and "tree" edge now become ambiguous; I used to
call them "ontological X edge", but I had fallen out of that habit; perhaps
I need to reintroduce that rigor.
DEV-13162
This modifies the xmlo reader, xmlo->AIR lowering, and AIR->ASG to introduce
a package for identifiers. It does not yet, however, add edges from the
package to the identifier.
Once edges are added, the DFS will change in undesirable ways, which will
require a new implementation. This is desirable to decouple from Petgraph
anyway, and then will be able to restore the prior single-pass sort+cycle
check.
That will also encapsulate visiting behavior within the `asg::graph` module
and, in turn, allow encapsulating `Asg.graph` finally.
DEV-13162
This doesn't go far enough, but it elaborates a bit---the existing was far
too much of a catch-all. It's important to take advantage of exhaustiveness
checks to ensure each transition is properly accounted for.
This parser is going to get more work over time, including right now, so I'm
not going to go too deep into this yet, but it's be useful (as a reader) to
compare it to e.g. asg::air's parsers' explicit enumeration of states and
favoring of explicit errors over dead state transitions.
DEV-13162
This may now index _any_ type of object, in preparation for indexing package
import paths. In practice, this only makes sense (at least currently) for
`Pkg` and `Ident`.
This generalization also applies to `Asg::lookup_or_missing`.
DEV-13162
Historically, the ASG was better described as a "dependency graph",
containing only identifiers (which are simply called "symbols" in the
XSLT-based compiler). Consequently, it was appropriate for the graph to
have operations specific to identifiers. (Indeed, that's the only type of
object the graph supported.)
Much has changed since then. This cleans things up, and makes parenting
identifiers to root an _explicit_ operation. This will make it easier to
move forward with handling of scope, and importing identifiers into
packages, and removing `Source`, and so on.
DEV-13162
I've been torturing myself trying to figure out how I want to generalize
indexing, lookups, and value numbering in a way that is appropriate for this
project (that is, not over-engineered relative to my needs).
Before I can do much of anything, though, I need to stop having indexing
only as a `Root` thing (previously it wasn't even tied to `Root`). This
makes that change for tamec, but temporarily removes scoping concerns until
I can add more specific types of indexing.
Not only does this allow cleaning up some `Ident`-specific stuff from `Asg`,
but the cleanup also helps to show that portions of the system aren't still
using Root-based globals.
The linker (`tameld`) still uses the old `global` methods for now; those
will eventually go away, but this needs to change to unify both tamec and
tameld once we get to imports as part of the compiler.
DEV-13162
This is needed to then support `@desc` for shorthand desugaring; it's
required by the XSLT-based compiler (and will eventually be required by
TAMER too).
DEV-13708
This is needed by TAMER's template desugaring. The XSD is superceded by
`nir::parse`, but can't go away until TAMER fully supplants the XSLT-based
compiler.
...and after all this time, I still never got rid of the duplicate XSD. Or
even recall which one is the duplicate.
DEV-13708
TAMER desugars shorthand template application bodies (`@values@`) into _the
name of a closed template_ whose body should be expanded into place. This
change recognizes that convention, and makes use of it.
Desugaring is part of `nir::tplshort`.
DEV-13708
XIRF->Nir produces `Todo` and `TodoAttr` tokens for many different
things. The previous approach was to ignore those things so that I could
begin adding portions of packages to the graph and observe how that goes.
But now that I'm starting to be able to compile certain packages that
utilize only small subsets of TAME features, I need to have confidence that
I'm fully parsing them. This means rejecting tokens that I haven't yet
gotten to.
DEV-13708
This supports arbitrary documentation as sibling text (mixed content, in XML
terms). The motivation behind this change is to permit existing system
tests to succeed when `Todo | TodoAttr` are both rejected, rather than
having to ignore this.
TAME has always had a philosophy of literate documentation, however it was
never fully realized. This just maintains the status quo; the text is
unstructured, and maybe will be parsed in the future.
Unfortunately, this does _not_ include the output in the `xmli` file or the
system tests. The reason has nothing to do with TAMER---`xmllint` does not
format the output when there is mixed content, it seems, and I need to move
on for now; I'll consider my options in the future. But, it's available on
the graph and ready to go.
DEV-13708
This _only_ re-introduces for PackageStmt since that's all I have tests for
at present. More will be re-added later.
They were previously removed when the attribute parsing was upended in
`ele_parse!`.
This does lose the attribute name, compared to before; that'll ideally be
re-added, and I'll explore options for doing so later, since I also want
them in other contexts. But it needs to be done generically (not
XML-related).
This had to be done before blowing up on TODOs, or system tests would fail.
DEV-13708
This is in preparation for throwing errors (with diagnostic information) on
yet-to-be-supported tokens, so that I can confidently compile individual
packages without worrying that something is just being ignored.
This makes obvious that `ele_parse!` had a different design in mind
previously, and it's now resulting in a lot of boilerplate; I'll address
that in the future once I'm certain requirements have been settled on, since
I've spent far too much time on it to waste more.
DEV-13708
This introduces a new `Doc` object that can be owned by `Expr` (only atm)
and contain what it describes as a concise independent clause. This
construction is not enforced, and is only really obvious today via the
Summary Pages.
There's a lot of latent and unrealized potential in TAME's documentation
philosophy that was never realized, so this will certainly evolve over
time. But for now, the primary purpose was to get `@desc` working on things
like classifications so that `xmli` output can compile for certain
packages.
DEV-13708
These are used by virtually every `ObjectKind`; I've been meaning to do this
for a while, but now that I'm about to introduce a new one (`Doc`), let's
just get it out of the way.
DEV-13708
This doesn't do the actual hard work yet of resolving and loading a package,
but it does place it on the graph and re-derive it into the xmli output.
DEV-13708
This introduces `<match on="foo" />` and `<match on="foo" value="bar" />`,
which are both equality predicates. Other types of predicates are not yet
supported.
This change is a bit messy and leaves a bit to be desired. `NirToAir` is
quite messy and needs some cleanup. There's also the issue of introducing
XML-specific errors in NIR so that users know what things like "subject"
mean, but not being able to do so yet because NIR is agnostic to the source
document type; another layer of abstraction is needed.
But, my priority is first to get derivation of a particularly
expensive (generated) package in our internal systems working first.
DEV-13708
The alternative I was floating was a tagged `Ref` (that is, with an enum
within it), but I settled on this for now, in part for a more concise
notation with the mapping in nir::parse.
We'll see how this evolves. For now, it's not important with the only thing
that uses ref in nir::parse, which is template application.
This was introduced for `match`, which is to come shortly.
DEV-13708
This recognizes template application within expressions. Since expressions
can occur within templates, this can occur arbitrarily deeply.
And with that, we have the core of the template system represented on the
graph. Of course, there are some glaring scoping issues to be resolved, but
those aren't unique to template application.
DEV-13708
I had hoped this would be considerably easier to implement, but there are
some confounding factors.
First of all: this accomplishes the initial task of getting nested template
applications and definitions re-output in the `xmli` file. But to do so
successfully, some assumptions had to be made.
The primary issue is that of scope. The old (XSLT-based) TAME relied on the
output JS to handle lexical scope for it at runtime in most situations. In
the case of the template system, when scoping/shadowing were needed, complex
and buggy XPaths were used to make a best effort. The equivalent here would
be a graph traversal, which is not ideal.
I had begun going down the rabbit hole of formalizing lexical scope for
TAMER with environments, but I want to get this committed and working first;
I've been holding onto this and breaking off changes for some time now.
DEV-13708
All ObjectIndex-like objects hash using only the underlying identifier,
which ultimately boils down to a `NodeIndex` (petgraph), which is just a
u32. And so in that sense, the only purpose we have for hashing it is to
(a) reduce the space required to store mappings, and (b) compose with other
`Hash`es.
DEV-13708
This creates another trait and struct `ObjectIndexToTree` that assert a
stronger invariant than `ObjectIndexRelTo`---that not only does it uphold
the invariants of `ObjectIndexRelTo`, but also that it represents a _tree_
edge, which indicates _ownership_ rather than just a reference.
This will be used to statically infer what can serve as a scope boundary for
upcoming changes. Specifically, anything that can own an `Ident` introduces
a new level of scope.
DEV-13708
This allows this method to be used on anything that is able to relate to an
identifier, which is needed for the changes being made for the template
system.
This linear lookup is actually going away (as hinted at by preceding
commits); this is extracted as part of a larger change and I wanted to get
it committed to make it easier to follow upcoming changes.
DEV-13708
The prior commit begins to explain the end goal of being able to index
identifiers outside of the global environment.
This change continues to index things as before, but introduces a new key
based on the pair of the symbol id together with a node that is _part of_
its target environment. The only environment utilized at the moment (in this
commit) is that of the root node (which is the global scope), in both
indexing and lookup. Future commits will extend this, and contain more
information about and rationale for the implementation.
The new general index methods are restricted to `pub(super)` until an
abstraction can be put in place that is responsible for environment
indexing; that's a responsibility that is currently handled by
`AirAggregateCtx` for tamec, and the linker has no scoping
requirements since all of that has already been dealt with.
DEV-13708
This reverts commit 1b7eac337cd5909c01ede3a5b3fba577898d5961.
This is a revert of the previous revert, just so that I (and you) have
references to prior rationale.
This was previously reverted because it wasn't worth doing, but now we have
a situation where we need to begin implementing lexical scoping rules for
nested containers (packages and templates). In particular, as you'll see in
the commits that follow, we need to be able to look up an identifier that
may have been created as Missing at one level of scope (certain types of
blocks), but then define it at another level.
Or, even more simply at this point, since I'm not yet doing anything
sophisticated with scope: we're only indexing in the global environment, and
we need to be able to index elsewhere too.
The next commit will go into more information, but suffice it to say for now
that indexing is going to get more complicated than a SymbolId.
Sticking with FxHash for now; we don't need a stable hash now.
DEV-13708
Mike Gerwitz