Compare commits

...

47 Commits

Author SHA1 Message Date
Mike Gerwitz d889aca13a tamer: asg::graph::AsgRelMut: API cleanup
This does two things:

  1. Removes callback; it didn't add anything of practical value.
     The operation will simply be performed as long as no error is provided
     by the callee.
  2. Consolodates three arguments into `ProposedRel`.  This makes blocks in
     `object_rel!` less verbose and boilerplate-y.

I'll probably implement `TplShape::Unknown` via the dynamic `Ident` `Tpl`
edge before continuing with any cleanup.  This is getting pretty close to
reasonable for future implementations.

DEV-13163
2023-07-27 03:33:06 -04:00
Mike Gerwitz 579575a358 tamer: asg::graph::object: ObjectIndex::try_map_obj_inner
Continuing to clean house and make things more concise, not just for `tpl`
but for all future changes.

DEV-13163
2023-07-27 03:07:33 -04:00
Mike Gerwitz 66512bf20d tamer: f: impl_mono_map! macro
This helps to remove some boilerplate.  Testing this out in
`asg::graph::object::tpl` before applying it to other things; really `Map`
can just go away entirely then since it can be implemented in terms of
`TryMap`, but maybe it should stick around for manual impls (implementing
`TryMap` manually is more work).

DEV-13163
2023-07-27 02:56:29 -04:00
Mike Gerwitz 3c9e1add20 tamer: f: Add TryMap
This implements TryMap and utilizes it in `asg::graph::object::tpl`.

DEV-13163
2023-07-27 01:44:12 -04:00
Mike Gerwitz 38c0161257 tamer: f::{Functor=>Map}: It's not really a functor
At least not how most people expect functors to be.  I'm really just using
this as a map with powerful inference properties that make writing code more
pleasent.

And I need fallible methods now too.

DEV-13163
2023-07-26 16:43:09 -04:00
Mike Gerwitz e14854a555 tamer: asg::air::tpl: Resolve shape against inner template application
Things are starting to get interesting, and this shows how caching
information about template shape (rather than having to query the graph any
time we want to discover it) makes it easy to compose shapes.

This does not yet handle the unknown case.  Before I do that, I'll want to
do some refactoring to address duplication in the `tpl` module.

DEV-13163
2023-07-26 16:09:17 -04:00
Mike Gerwitz c19ecba6ef tamer: asg::air::object::tpl: Reject multi-expression shape
This enforces the new constraint that templates expanding into an `Expr`
context must only inline a single `Expr`.

Perhaps in the future we'll support explicit splicing, like `,@` in
Lisp.  But this new restriction is intended for two purposes:

  - To make templates more predictable (if you have a list of expressions
    inlined then they will act differently depending on the type of
    expression that they are inlined into, which means that more defensive
    programming would otherwise be required); and
  - To make expansion easier, since we're going to have to set aside an
    expansion workspace ahead of time to ensure ordering (Petgraph can't
    replace edges in-place).  If we support multi-expansion, we'd have to
    handle associativity in all expression contexts.

This'll become more clear in future commits.

It's nice to see all this hard work coming together now, though; it's easy
now to perform static analysis on the system, and any part of the graph
construction can throw errors with rich diagnostic information and still
recover properly.  And, importantly, the system enforces its own state, and
the compiler helps us with that (the previous commits).

DEV-13163
2023-07-26 04:03:52 -04:00
Mike Gerwitz aec721f4fa tamer: asg::graph: Work AsgRelMut specialization into `object_rel!` macro
This formalizes the previous commit a bit more and adds documentation
explaining why it exists and how it works.  Look there for more
information.

This has been a lot of setup work.  Hopefully things are now easier in the
future.  And now we have nice declarative type-level hooks into the graph!

DEV-13163
2023-07-26 04:03:49 -04:00
Mike Gerwitz 37c962a7ee tamer: asg::graph::object::tpl::TplShape: Introduce template "shapes"
This change is the first to utilize matching on edges to determine the state
of the template (to begin to derive its shape).

But this is notable for my finally caving on `min_specialization`.

The commit contains a bunch of rationale for why I introduced it.  I've been
sitting on trying it for _years_.  I had hoped for further progress in
determining a stabalization path, but that doesn't seem to be happening.

The reason I caved is because _not_ using it is a significant barrier to
utilizing robust types in various scenarios.  I've been having to work
around that with significant efforts to write boilerplate code to match on
types and branch to various static paths accordingly.  It makes it really
expensive to make certain types of changes, and it make the code really
difficult to understand once you start to peel back abstractions that try to
hide it.

I'll see how this goes and, if it goes well, begin to replace old methods
with specialization.

See the next commit for some cleanup.  I purposefully left this a bit of a
mess (at the bottom of `asg::graph::object::tpl`) to emphasize what I'm
doing and why I introduced it.

DEV-13163
2023-07-25 15:28:53 -04:00
Mike Gerwitz 4168c579fd tamer: asg::graph::Asg{Object=>Rel}Mut: Trait-level target type
This allows for a declarative matching on edge targets using the trait
system, rather than having to convert the type to a runtime value to match
on (which doesn't make a whole lot of sense).

See a commit to follow shortly (with Tpl) for an example use case.

DEV-13163
2023-07-25 11:15:41 -04:00
Mike Gerwitz 2ecc143e02 tamer: asg::graph::pre_add_edge: Use pre-narrowed source ObjectIndex
Since we're statically invoking a particular ObjectKind's method, we already
know the source type.  Let's pre-narrow it for their (my) convenience.

DEV-13163
2023-07-25 10:55:08 -04:00
Mike Gerwitz 28b83ad6a3 tamer: asg::graph::AsgObjectMut: Allow objects to assert ownership over relationships
There's a lot to say about this; it's been a bit of a struggle figuring out
what I wanted to do here.

First: this allows objects to use `AsgObjectMut` to control whether an edge
is permitted to be added, or to cache information about an edge that is
about to be added.  But no object does that yet; it just uses the default
trait implementation, and so this _does not change any current
behavior_.  It also is approximately equivalent cycle-count-wise, according
to Valgrind (within ~100 cycles out of hundreds of millions on large package
tests).

Adding edges to the graph is still infallible _after having received
permission_ from an `ObjectIndexRelTo`, but the object is free to reject the
edge with an `AsgError`.

As an example of where this will be useful: the template system needs to
keep track of what is in the body of a template as it is defined.  But the
`TplAirAggregate` parser is sidelined while expressions in the body are
parsed, and edges are added to a dynamic source using
`ObjectIndexRelTo`.  Consequently, we cannot rely on a static API to cache
information; we have to be able to react dynamically.  This will allow `Tpl`
objects to know any time edges are added and, therefore, determine their
shape as the graph is being built, rather than having to traverse the tree
after encountering a close.

(I _could_ change this, but `ObjectIndexRelTo` removes a significant amount
of complexity for the caller, so I'd rather not.)

I did explore other options.  I rejected the first one, then rejected this
one, then rejected the first one again before returning back to this one
after having previously sidelined the entire thing, because of the above
example.  The core point is: I need confidence that the graph isn't being
changed in ways that I forgot about, and because of the complexity of the
system and the heavy refactoring that I do, I need the compiler's help;
otherwise I risk introducing subtle bugs as objects get out of sync with the
actual state of the graph.

(I wish the graph supported these things directly, but that's a project well
outside the scope of my TAMER work.  So I have to make do, as I have been
all this time, by layering atop of Petgraph.)

(...I'm beginning to ramble.)

(...beginning?)

Anyway: my other rejected idea was to provide attestation via the
`ObjectIndex` APIs to force callers to go through those APIs to add an edge
to the graph; it would use sealed objects that are inaccessible to any
modules other than the objects, and assert that the caller is able to
provide a zero-sized object of that sealed type.

The problem with this is...exactly what was mentioned above:
`ObjectIndexRelTo` is dynamic.  We don't always know the source object type
statically, and so we cannot make those static assertions.

I could have tried the same tricks to store attestation at some other time,
but what a confusing mess it would be.

And so here we are.

Most of this work is cleaning up the callers---adding edges is now fallible,
from the `ObjectIndex` API standpoint, and so AIR needed to be set up to
handle those failures.  There _aren't_ any failures yet, but again, since
things are dynamic, they could appear at any moment.  Furthermore, since
ref/def is commutative (things can be defined and referenced in any order),
there could be surprise errors on edge additions in places that might not
otherwise expect it in the future.  We're now ready for that, and I'll be
able to e.g. traverse incoming edges on a `Missing->Transparent` definition
to notify dependents.

This project is going to be the end of me.  As interesting as it is.

I can see why Rust just chose to require macro definitions _before_ use.  So
much less work.

DEV-13163
2023-07-24 16:41:32 -04:00
Mike Gerwitz e414782def tamer: asg::graph: Encapsulate edge additions
AIR is no longer able to explicitly add edges without going through an
object-specific `ObjectIndex` API.  `Asg::add_edge` was already private, but
`ObjectIndex::add_edge_{to,from}` was not.

The problem is that I want to augment the graph with other invariants, such
as caches.  I'd normally have this built into the graph system itself, but I
don't have the time for the engineering effort to extend or replace
Petgraph, so I'm going to build atop of it.

To have confidence in any sort of caching, I need assurances that the graph
can't change out from underneath an object.  This gets _close_ to
accomplishing that, but I'm still uncomfortable:

  - We're one `pub` addition away from breaking these invariants; and
  - Other `Object` types can still manipulates one-anothers' edges.

So this is a first step that at least proves encapsulation within
`asg::graph`, but ideally we'd have the system enforce, statically, that
`Objects` own their _outgoing_ edges, and no other `Object` is able to
manipulate them.  This would ensure that any accidental future changes, or
bugs, will cause compilation failures rather than e.g. allowing caches to
get out of sync with the graph.

DEV-13163
2023-07-21 10:21:57 -04:00
Mike Gerwitz 0f93f3a498 tamer: NIR->xmli interpolation and template param
The fixpoint tests for `meta-interp` are finally working.  I could have
broken this up more, but I'm exhausted with this process, so, you get what
you get.

NIR will now recognize basic `<text>` and `<param-value>` nodes (note the
caveat for `<text>` in the comment, for now), and I finally include abstract
binding in the lowering pipeline.  `xmli` output is also now able to cope
with metavariables with a single lexical association, and continues to
become more of a mess.

DEV-13163
2023-07-18 12:31:28 -04:00
Mike Gerwitz 85b08eb45e tamer: nir::interp: Do not include original specification in generated desc
This is a really obvious problem in retrospect, which makes me feel rather
silly.

The output was useful, but I don't have time to deal with this any further
right now.  The comments in the commit explain the problem---that the output
ends up being interpolated as part of the fixpoint test, in an incorrect
context, and so the code that we generate is invalid.  Also goes to show why
the fixpoint tests are important.

(Yes, they're still disabled for meta-interp, I'm trying to get them
enabled.)

DEV-13163
2023-07-18 11:17:51 -04:00
Mike Gerwitz 507669cb30 tamer: asg::graph::object::ObjectIndexRefined: New narrowing type
The provided documentation provides rationale, and the use case is the
ontree change.  I was uncomfortable without the exhaustive match, and I was
further annoyed by the lack of easy `ObjectIndex` narrowing.

DEV-13163
2023-07-18 10:31:33 -04:00
Mike Gerwitz 5a301c1548 tamer: asg::graph::visit::ontree: Source ordering of ontological tree
This introduces the ability to specify an edge ordering for the ontological
tree traversal.  `tree_reconstruction` will now use a
`SourceCompatibleTreeEdgeOrder`, which will traverse the graph in an order
that will result in a properly ordered source reconstruction.  This is
needed for template headers, because interpolation causes
metavariables (exposed as template params) to be mixed into the body.

There's a lot of information here, including some TODOs on possible
improvements.  I used the unstable `is_sorted` to output how many template
were already sorted, based on one of our very large packages internally that
uses templates extensively, and found that none of the desugared shorthand
template expansions were already ordered.  If I tweak that a bit, then
nearly all templates will already be ordered, reducing the work that needs
to be done, leaving only template definitions with interpolation to be
concerned about, which is infrequent relative to everything else.

DEV-13163
2023-07-18 10:31:31 -04:00
Mike Gerwitz b30018c23b tamer: xmli reconstruction of desugared interpolated metavars
Well, this is both good news and bad news.

The good news is that this finally produces the expected output and
reconstructs sources from interpolated values on the ASG.  Yay!

...the bad news is that it's wrong.  Notice how the fixpoint test is
disabled.

So, my plan was originally to commit it like this first and see if I was
comfortable relaxing the convention that `<param>` nodes had to appear in
the header.  That's nice to do, that's cleaner to do, but would the
XSLT-based compiler really care?  I had to investigate.

Well, turns out that TAMER does care.  Because, well over a decade ago, I
re-used `<param>`, which could represent not only a template param, but also
a global param, or a function param.

So, XML->NIR considers all `<param>` nodes at the head of a template to be
template parameters.  But after the first non-header element, we transition
to another state that allows it to be pretty much anything.

And so, I can't relax that restriction.

And because of that, I can't just stream the tree to the xmli generator,
I'll have to queue up nodes and order them.

Oh well, I tried.

DEV-13163
2023-07-17 14:20:05 -04:00
Mike Gerwitz 85892caeb2 tamer: asg: Root abstract identifiers in active container
I'm not sure how I overlooked this previously, and I didn't notice until
trying to generate xmli output.  I think I distracted myself with the
use of dangling status, which was not appropriate, and that has since
changed so that we have a dedicated concept.

This introduces the term "instantiation", or more specifically "lexical
instantiation".  This is more specific and meaningful than simply
"expansion", which is what occurs during instantiation.  I'll try to adjust
terminology and make things more consistent as I go.

DEV-13163
2023-07-17 14:20:04 -04:00
Mike Gerwitz 760223f0c9 tamer: asg::air: Extract abstract definition into context
This logic ought to live alongside other definition logic...which in turn
needs its own extraction, but that's a separate concern.

This makes the definition of abstract identifiers very similar to
concrete.  But, treating these as dangling, even if that's technically true,
has to change---we still want an edge drawn to the abstract identifier via
e.g. a template since we want the graph to mirror the structure of what it
will expand into concretely.  I didn't notice this problem until trying to
generate the xmli for it.

So, see the commit to follow.

DEV-13163
2023-07-13 11:16:10 -04:00
Mike Gerwitz b4b85a5e85 tamer: asg::air: Support Meta::ConcatList with lexemes and refs
This handles the common cases for meta, which includes what interpolation
desugars into.  Most of this work was in testing and reasoning about the
issue; `asg::graph::visit:ontree::test` has a good summary of the structure
of the graph that results.

The last remaining steps to make this work end-to-end is for NIR->AIR to
lower `Nir::Ref` into `Air::BindIdent`, and then for `asg::graph::xmli` to
reconstruct concatenation lists.  I'll then be able to commit the xmli test
case I've been sitting on, whose errors have been guiding my development.

DEV-13163
2023-07-13 10:48:45 -04:00
Mike Gerwitz d2d29d8957 tamer: asg::air::meta: Use term "metalinguistic" over "metasyntactic"
The term "metasyntactic" made sense literally---it's a variable in a
metalanguage that expands into a context that is able to contribute to the
language's syntax.  But, the term has a different conventional use in
programming that is misleading.

The term "metalinguistic" is used in mathematics, to describe a metalanguage
or schema atop of a language.  This is more fitting.

DEV-13163
2023-07-13 10:48:45 -04:00
Mike Gerwitz b4bbc0d8f0 tamer: asg::air: Use new parse::util::spair function to reduce test ceremony
This makes `SPair` construction more concise, getting rid of the `into`
invocations.  For now I have only made this change in AIR's tests, since
that's what I'm working on and I want to observe how this convention
evolves.  This may also encourage other changes, e.g. placing spans within
the `toks` array, rather than having to jump around the test for them.

The comment for `spair` mentions why this is a test-only function.  But it
also shows how dangerous `impl Into<SymbolId> for &str` can be, since it
seems so innocuous---it uses a global interner.  I'll be interested to see a
year from now if I decided to forego that impl in favor of explicit
internment, since I'm not sure it's worth the convenience anymore.

DEV-13163
2023-07-13 10:48:44 -04:00
Mike Gerwitz 8a10f8bbbe tamer: asg::air: Remove unncessary vec![] usage in tests
This has been bothering me for quite a long time, and is just more test
cleanup before I introduce more.  I suspect this came from habit with the
previous Rust edition where `into_iter()` on arrays was a much more verbose
operation.

To be clear: this change isn't for performance.  It's about not doing
something silly when it's unnecessary, which also sets a bad example for
others.

There are many other tests in other modules that will need updating at some
point.

DEV-13163
2023-07-13 10:48:27 -04:00
Mike Gerwitz 2e33e9e93e tamer: asg::air: Remove `Air::` token variant prefixes from tests
This just removes noise from test, as has become standard in various other
tests in TAMER.

DEV-13163
2023-07-13 10:48:27 -04:00
Mike Gerwitz 24ee041373 tamer: asg::air: Support abstract biding of `Expr`s
This produces a representation of abstract identifiers on the graph, for
`Expr`s at least.  The next step will probably be to get this working
end-to-end in the xmli output before extending it to the other remaining
bindable contexts.

DEV-13163
2023-07-13 10:48:26 -04:00
Mike Gerwitz a144730981 tamer: nir::abstract_bind: Require @-padding of metavariable names
This enforces the naming convention that is utilized to infer whether an
identifier binding must be translated to an abstract binding.

This does not yet place any restrictions on other characters in identifier
names; both the placement of and flexibility of that has yet to be
decided.  This change is sufficient enough to make abstract binding
translation reliable.

DEV-13163
2023-07-10 10:28:01 -04:00
Mike Gerwitz 8449a2b759 tamer: parse::prelude: Include Display, Debug, and Error-related exports
Cut down on the import boilerplate some more for `ParseState`s.

DEV-13163
2023-07-10 10:27:57 -04:00
Mike Gerwitz 8685527feb tamer: nir: New token BindIdentMeta
The previous commit made me uncomfortable; we're already parsing with great
precision (and effort!) the grammar of NIR, and know for certain whether
we're in a metavariable binding context, so it makes no sense to have to try
to guess at another point in the lowering pipeline.

This introduces a new token to retain that information from XIR->NIR
lowering and then re-simplifies the lowering operation that was just
introduced in the previous commit (`AbstractBindTranslate`).

DEV-13163
2023-06-28 09:48:15 -04:00
Mike Gerwitz 7314562671 tamer: nir::abstract_bind: New lowering operation
This builds upon the concepts of the previous commit to translate identifier
binding into an abstract binding if it utilizes a symbol that follows a
metavariable naming convention.

See the provided documentation for more information.

This commit _does not_ integrate this into the lowering pipeline yet, since
the abstract identifiers are still rejected (as TODOs) by AIR.

DEV-13163
2023-06-28 09:48:14 -04:00
Mike Gerwitz 15071a1824 tamer: nir: Interpolate concrete binds into abstract binds
This introduces the notion of an abstract identifier, where the previous
identifiers are concrete.  This serves as a compromise to either introducing
a new object type (another `Ident`), or having every `Ident` name be defined
by a `Meta` edge, which would bloat the graph significantly.

This change causes interpolation within a bind context to desugar into a new
`BindIdentAbstract` token, but AIR will throw an error if it encounters it
for now; that implementation will come soon.

This does not yet handle non-interpolation cases,
e.g. `<classify as="@foo@">`.  This is a well-established shorthand for
`as="{@foo@}"`, but is unfortunately ambiguous in the context of
metavariable definitions (template parameters).  This language ambiguity
will have to be handled here, and will have to fall back to today's behavior
of assuming concrete in that `param/@name` context but abstract every else,
unless of course interpolation is triggered using `{}` to disambiguate (as
in `<param name="{@foo@}"`).

I was going to handle the short-hand meta binding case as part of
interpolation, but I decided it may be appropriate for its own lowering
operation, since it is intended to work regardless of whether interpolation
takes place; it's a _translation_ of a binding into an abstract one, and it
can clearly delineate the awkward syntactic rules that we have to inherit,
as mentioned above.

DEV-13163
2023-06-27 12:48:19 -04:00
Mike Gerwitz 828d8918a3 tamer::asg::graph::object::ident::Ident::name: Produce Option
This prepares to make the name of an `Ident` optional to support abstract
identifiers derived from metavariables.

This is an unfortunate change to have to prepare for, since it complicates
how Idents are interpreted, but the alternative (a new object type) is not
good either.  We'll see how this evolves.

DEV-13163
2023-06-26 15:37:08 -04:00
Mike Gerwitz 6b54eafd70 tamer: asg::air: Hoist metavars in expressions
This is intended to support NIR's lexical interpolation, which expands in
place into metavariables.

This commit does not yet contain the NIR portion (or xmli system test)
because Meta needs to be able to handle concatenation first; that's next.

DEV-13163
2023-06-20 15:14:38 -04:00
Mike Gerwitz d10bf00f5d tamer: Initial template/param support through xmli
This introduces template/param and regenerates it in the xmli output.  Note
that this does not check that applications reference known params; that's a
later phase.

DEV-13163
2023-06-14 16:38:05 -04:00
Mike Gerwitz 9887abd037 tamer: nir::air: Include mention of .experimental file in TODO help
The previous commit introduced support for a `.experimental` file to tigger
`xmlo-experimental`.  This modifies the error message for unsupported
features to make mention of it to help to the user track down the problem.

DEV-13162
2023-06-14 13:03:21 -04:00
Mike Gerwitz a9bbb87612 build-aux/Makefile.am: Introduce .experimental files
If a source file is paired with a `.experimental` file (for example,
`foo.xml` has a silbing `foo.experimental` file), then it will be
precompiled using `--emit xmlo-experimental` instead of `--emit
xmlo`.  Further, the contents of the experimental file may contain
`#`-prefixed comments describing why it exists, as well as additional
options to pass to `tamec`.

For example, if this is an experimental file:

```

--foo
--bar=baz
```

Then the tamec invocation will contain:

  tamec [...] --emit xmlo-experimental --foo --bar=baz -o foo.xmli

This allows for package-level conditional compilation with new features so
that I am able to focus on packages that will provide the most meaningful
benefits to our team, whether they be performance or features.

DEV-13162
2023-06-14 12:02:57 -04:00
Mike Gerwitz 7487bdccc3 tamer: nir::air: Recoverable error instead of panic for TODO tokens
Now that the feature flag for the parser is a command line option, it is
useful to be able to run it on any package and see what errors arise, to use
as a guide for development with the goal of getting a particular package to
compile.

This converts the TODO panic into a recoverable error so that the parser can
spit out as many errors as it can.

DEV-13162
2023-06-14 10:24:50 -04:00
Mike Gerwitz 454f5f4d04 tamer: Initial clarifying pipeline docs
This provides some initial information to help guide a user to discover how
TAMER works, though either the source code or the generated
documentation.  This will improve over time, since all of the high-level
abstractions are still under development.

DEV-13162
2023-06-13 23:43:04 -04:00
Mike Gerwitz 9eeb18bda2 tamer: Replace wip-asg-derived-xmli flag with command line option
This introduces `xmlo-experimental` for `--emit`, allowing the new parser to
be toggled selectively for individual packages.  This has a few notable
benefits:

  1. We'll be able to conditionally compile packages as they are
     supported (TAMER will target specific packages in our system to try to
     achieve certain results more quickly);

  2. This cleans up the code a bit by removing awkward gated logic, allowing
     natural abstractions to form; and

  3. Removing the compile-time feature flag ensures that the new features
     are always built and tested; there are fewer configuration combinations
     to test.

DEV-13162
2023-06-13 23:23:51 -04:00
Mike Gerwitz 341af3fdaf tamer: nir::air: Dynamic configuration in place of static wip-asg-derived-xmli flag
This flag should have never been sprinkled here; it makes the system much
harder to understand.

But, this is working toward a command-line tamec option to toggle NIR
lowering on/off for various packages.

DEV-13162
2023-06-13 15:07:03 -04:00
Mike Gerwitz 61d556c89e tamer: pipeline: Generate recoverable sum error types
This was a significant undertaking, with a few thrown-out approaches.  The
documentation describes what approach was taken, but I'd also like to
provide some insight into the approaches that I rejected for various
reasons, or because they simply didn't work.

The problem that this commit tries to solve is encapsulation of error
types.

Prior to the introduction of the lowering pipeline macro
`lower_pipeline!`, all pipelines were written by hand using `Lower` and
specifying the applicable types.  This included creating sum types to
accommodate each of the errors so that `Lower` could widen automatically.

The introduction of the `lower_pipeline!` macro resolved the boilerplate and
type complexity concerns for the parsers by allowing the pipeline to be
concisely declared.  However, it still accepted an error sum type `ER` for
recoverable errors, which meant that we had to break a level of
encapsulation, peering into the pipeline to know both what parsers were in
play and what their error types were.

These error sum types were also the source of a lot of tedious boilerplate
that made adding new parsers to the pipeline unnecessarily unpleasant;
the purpose of the macro is to make composition both easy and clear, and
error types were undermining it.

Another benefit of sum types per pipeline is that callers need only
aggregate those pipeline types, if they care about them, rather than every
error type used as a component of the pipeline.

So, this commit generates the error types.  Doing so was non-trivial.

Associated Types and Lifetimes
------------------------------
Error types are associated with their `ParseState` as
`ParseState::Error`.  As described in this commit, TAMER's approach to
errors is that they never contain non-static lifetimes; interning and
copying are used to that effect.  And, indeed, no errors in TAMER have
lifetimes.

But, some `ParseState`s may.  In this case, `AsgTreeToXirf`:

```
impl<'a> ParseState for AsgTreeToXirf<'a> {
  // [...]
  type Error = AsgTreeToXirfError;
  // [...]
}
```

Even though `AsgTreeToXirfError` does not have a lifetime, the `ParseState`
it is associated with _does_`.  So to reference that type, we must use
`<AsgTreeToXirf<'a> as ParseState>::Error`.  So if we have a sum type:

```
enum Sum<'a> {
  //     ^^ oh no!                  vv
  AsgTreeToXirfError(<AsgTreeToXirf<'a> as ParseState>::Error),
}
```

There's no way to elide or make anonymous that lifetime, since it's not
used, at the time of writing.  `for<'a>` also cannot be used in this
context.

The solution in this commit is to use a macro (`lower_error_sum`) to rewrite
lifetimes: to `'static`:

```
enum Sum {
  AsgTreeToXirfError(<AsgTreeToXirf<'static> as ParseState>::Error),
}
```

The `Error` associated type will resolve to `AsgTreeToXirfError` all the
same either way, since it has no lifetimes of its own, letalone any
referencing trait bounds.

That's not to say that we _couldn't_ support lifetimes as long as they're
attached to context, but we have no need to at the moment, and it adds
significant cognitive overhead.  Further, the diagnostic system doesn't deal
in lifetimes, and so would need reworking as well.  Not worth it.

An alternative solution to this that was rejected is an explicitly `Error`
type in the macro application:

```
// in the lowering pipeline
|> AsgTreeToXirf<'a> {  // lifetime
    type Error = AsgTreeToXirfError;   // no lifetime
}
```

But this requires peeling back the `ParseState` to see what its error is and
_duplicate_ it here.  Silly, and it breaks encapsulation, since the lowering
pipeline is supposed to return its own error type.

Yet another option considered was to standardize a submodule convention
whereby each `ParseState` would have a module exporting `Error`, among other
types.  This would decouple it from the parent type.  However, we still have
the duplication between that and an associated type.  Further, there's no
way to enforce this convention (effectively a module API)---the macro would
just fail in obscure ways, at least with `macro_rules!`.  It would have been
an ugly kluge.

Overlapping Error Types
-----------------------
Another concern with generating the sum type, resolved in a previous commit,
was overlapping error types, which prohibited `impl From<E> for ER`
generation.

The problem with that a number of `ParseState`s used `Infallible` as their
`Error` type.  This was resolved in a previous commit by creating
Infallible-like newtypes (variantless enums).

This was not the only option.  `From` fits naturally into how TAMER handles
sum types, and fits naturally into `Lower`'s `WidenedError`.  The
alternative is generating explicit `map_err`s in `lower_pipeline!`.  This
would have allowed for overlapping error types because the _caller_ knows
what the correct target variant is in the sum type.

The problem with an explicit `map_err` is that it places more power in
`lower_pipeline!`, which is _supposed_ to be a macro that simply removes
boilerplate; it's not supposed to increase expressiveness.  It's also not
fun dealing with complexity in macros; they're much more confusing that
normal code.

With the decided-upon approach (newtypes + `From`), hand-written `Lower`
pipelines are just as expressive---just more verbose---as `lower_pipeline!`,
and handles widening for you.  Rust's type system will also handle the
complexity of widening automatically for us without us having to reason
about it in the macro.  This is not always desirable, but in this case, I
feel that it is.
2023-06-13 14:49:43 -04:00
Mike Gerwitz 31f6a102eb tamer: pipeline::macro: Partially applied pipeline
This configures the pipeline and returns a closure that can then be provided
with the source and sink.

The next obvious step would be to curry the source and sink.

But I wanted to commit this before I take a different (but equivalent)
approach that makes the pipeline operations more explicit and helps to guide
the user (developer) in developing and composing them.  The FP approach is
less boilerplate, but is also more general and provides less
guidance.  Given that composition at the topmost levels of the system,
especially with all the types involved, is one of the most confusing aspects
of the system---and one of the most important to get right and make clear,
since it's intended to elucidate the entire system at a high level, and
guide the reader.  Well, it does a poor job at that now, but that's the
ultimate goal.

In essence---brutally general abstractions make sense at lower levels, but
the complexity at higher levels benefits from rigid guardrails, even though
it does not necessitate it.

DEV-13162
2023-06-13 10:02:51 -04:00
Mike Gerwitz 26c4076579 tamer: obj::xmlo::reader: Emit token after symbol dependencies
This will allow a tamec xmlo reading pipeline to stop before fragment
loading.

DEV-13162
2023-06-12 12:37:12 -04:00
Mike Gerwitz 0b9e91b936 tamer: obj::xmlo::reader::XmloReader: Remove generics
This cleanup is an interesting one, because I think the present me may
disagree with the past me.

The use of generics here to compose the parser from smaller parsers was due
to how I wrote my object-oriented code in other languages: where a class was
an independently tested unit.  I was trying to reproduce the same here,
utilizing generics in the same way that one would use compoisition via
object constructors in other languages.

But it's been a long time since then, and I've come to settle on different
standards in Rust.  The components of `XmloReader` really are just
implementation details.  As I find myself about to want to modify its
behavior, I don't _want_ to compose `XmloReader` from _different_ parsers;
that may result in an invalid parse.  There's one correct way to parse an
xmlo file.

If I want to parse the file differently, then `XmloReader` ought to expose
a way of doing so.  This is more rigid, but that rigidity buys us confidence
that the system has been explicitly designed to support those
operations.  And that confidence gives us peace of mind knowing that the
system won't compose in ways that we don't intend for it to.

Of course, I _could_ design the system to compose in generic ways.  But
that's an over-generalization that I don't think will be helpful; it's not
only a greater cognitive burden, but it's also a lot more work to ensure
that invariants are properly upheld and to design an API that will ensure
that parsing is always correct.  It's simply not worth it.

So, this makes `XmloReader` consistent with other parsers now, like
`AirAggregate` and nir::parse (ele_parse).  This prepares for a change to
make `XmloReader` configurable to avoid loading fragments from object files,
since that's very wasteful for `tamec`.

DEV-13162
2023-06-12 12:37:12 -04:00
Mike Gerwitz 1bb25b05b3 tamer: Newtypes for all Infallible ParseState errors
More information will be presented in the commit that follows to generalize
these, but this sets the stage.

The recently-introduced pipeline macro takes care of most of the job of a
declarative pipeline, but it's still leaky, since it requires that the
_caller_ create error sum types.  This not only exposes implementation
details and so undermines the goal of making pipelines easy to declare and
compose, but it's also one of the last major components of boilerplate for
the lowering pipeline.

My previous attempts at generating error sum types automatically for
pipelines ran into a problem because of overlapping `impl`s for the various
`<S as ParseState>::Error` types; this resolves that issue via
newtypes.  I had considered other approaches, including explicitly
generating code to `map_err` as part of the lowering pipeline, but in the
end this is the easier way to reason about things that also keeps manual
`Lower` pipelines on the same level of expressiveness as the pipeline macro;
I want to restrict its unique capabilities as much as possible to
elimination of boilerplate and nothing more.

DEV-13162
2023-06-12 12:33:22 -04:00
Mike Gerwitz 672cc54c14 compiler/js.xsl: Derive supplier name from base package name
At or around 00492ace01, I modified packages
to output canonical `@name`s, which contains a leading forward
slash.  Previously, names omitted that slash.  I did not believe that this
caused any problems.

It seems that the XSLT-based `standalones` system utilizes this package name
to derive a supplier name, which is supposed to be the filename of the
package without any path.  Since the package name changed from
`suppliers/foo` to `/suppliers/foo`, for example, this was now producing
"suppliers/name" instead of "name".

Of course, it was never a good idea to strip off only the first path
component.  But, this is how it has been since TAME was originally created
well over a decade ago.

I did not catch this since I was diff'ing the output of the xmle files, not
the final JS files.  I had thought that was sufficient, given what I was
changing, but I was wrong.

DEV-14502
2023-06-08 16:46:18 -04:00
Mike Gerwitz d6e9ec7207 tamer: nightly pin: Describe problems with adt_const_param's ConstParamTy
See commit for description of the problem, describing why I'm not yet
upgrading to a currently nightly version.

DEV-14476
2023-06-06 11:00:44 -04:00
78 changed files with 5755 additions and 1398 deletions

View File

@ -136,7 +136,12 @@ common: $(xmlo_common)
program-ui: ui/package.strip.js ui/Program.js ui/html/index.phtml
# Handle an intermediate step as we transition to the new compiler
# Handle an intermediate step as we transition to the new compiler.
# If a source file is paired with an `*.experimental` file with the same
# stem, then it will trigger compilation using `xmlo-experimental`. The
# file may contain additional arguments to the pass to the compiler.
%.xmli: %.xml %.experimental
$(path_tame)/tamer/target/release/tamec --emit xmlo-experimental $$(grep -v '^#' $*.experimental) -o $@ $<
%.xmli: %.xml
$(path_tame)/tamer/target/release/tamec --emit xmlo -o $@ $<

View File

@ -159,7 +159,7 @@
<!-- make the name of the supplier available -->
<text>/**@expose*/rater.supplier = '</text>
<value-of select="substring-after( $name, '/' )" />
<value-of select="( tokenize( $name, '/' ) )[ last() ]" />
<text>'; </text>
<text>/**@expose*/rater.meta = meta;</text>

View File

@ -53,9 +53,3 @@ unicode-width = "0.1.5"
# This is enabled automatically for the `test` profile.
parser-trace-stderr = []
# Derive `xmli` file from the ASG rather than a XIR token stream. This
# proves that enough information has been added to the graph for the entire
# program to be reconstructed. The `xmli` file will be a new program
# _derived from_ the original, and so will not match exactly.
wip-asg-derived-xmli = []

View File

@ -32,6 +32,24 @@
# }
# }
#
# But edges also support arbitrary code definitions:
#
# object_rel! {
# Source -> {
# tree TargetA {
# fn pre_add_edge(...) {
# // ...
# }
# },
# tree TargetB,
# cross TargetC,
# }
# }
#
# And because of that,
# this script hard-codes the expected level of nesting just to make life
# easier.
#
# This script expects to receive a list of files containing such
# definitions.
# It will output,
@ -81,7 +99,8 @@ BEGINFILE {
/^object_rel! {$/, /^}$/ { in_block = 1 }
# `Foo -> {` line declares the source of the relation.
in_block && /->/ {
# We hard-code the expected depth to simplify parsing.
in_block && /^ \w+ -> \{$/ {
block_src = $1
printf " # `%s` from `%s:%d`\n", block_src, FILENAME, FNR
@ -92,7 +111,7 @@ in_block && /->/ {
# A closing curly brace always means that we've finished with the current
# source relation,
# since we're at the innermost level of nesting.
block_src && /}/ {
block_src && /^ }$/ {
block_src = ""
print ""
}
@ -116,7 +135,12 @@ block_src && /^ *\/\/ empty$/ {
# we must independently define each one.
# But that's okay;
# the output is quite legible.
block_src && $NF ~ /\w+,$/ {
block_src && /^ \w+ \w+(,| \{)$/ {
# Clean up the end of the string _before_ pulling information out of
# fields,
# since the number of fields can vary.
gsub(/(,| \{)$/, "")
# Edge type (cross, tree)
ty = $(NF-1)
@ -139,8 +163,6 @@ block_src && $NF ~ /\w+,$/ {
break;
}
gsub(/,$/, "")
# This may need updating over time as object names in Rust sources
# exceed the fixed-width definition here.
# This output is intended to form a table that is easy to read and

View File

@ -36,7 +36,7 @@
# you will need to modify `configure.ac` / `Makefile.am` to do your bidding.
[toolchain]
channel = "nightly-2023-04-15"
channel = "nightly-2023-04-15"
# The components should be checked in `configure.ac`
# - Note that `cargo-fmt` is `rustfmt`.
@ -50,3 +50,26 @@ components = ["rustfmt", "clippy"]
# another version of Rust already available elsewhere), you may use the
# `TAMER_RUST_TOOLCHAIN` `configure` parameter.
# ¹ TAMER uses the incomplete feature `adt_const_params`. "Incomplete"
# features require a special flag to enable---`incomplete_features`---and
# are described as "incomplete and may not be safe to use and/or cause
# compiler crashes".
#
# The `ConstParamTy` trait was introduced by
# <https://github.com/rust-lang/rust/pull/111670> and merged in early
# June 2023. After this change, const params types must implement this
# trait.
#
# Unfortunately, at the time of writing (2023-06), while the trait is
# implemented on a number of core primitives, it is _not_ implemented on the
# `NonZero*` types. There is an inquiry into this limitation on Zulip, and
# it is expected to be resolved in the future:
# <https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/ConstParamTy.20for.20NonZero*.20types>
#
# We will sit on this and watch how it evolves over the following
# weeks/months to assess how to best proceed.
#
# Aside from this, there seems to be a rustdoc bug at some point between
# April and June that causes documentation failures. This needs further
# investigation, and possibly a bug report.

View File

@ -36,7 +36,7 @@ use super::{
use crate::{
diagnose::Annotate,
diagnostic_unreachable,
f::Functor,
f::Map,
parse::{prelude::*, StateStack},
span::Span,
sym::SymbolId,
@ -46,6 +46,9 @@ use std::{
fmt::{Debug, Display},
};
#[cfg(test)]
use super::graph::object::ObjectRelTo;
#[macro_use]
mod ir;
use fxhash::FxHashMap;
@ -223,13 +226,14 @@ impl ParseState for AirAggregate {
(PkgTpl(tplst), AirBind(ttok)) => ctx.proxy(tplst, ttok),
(PkgTpl(tplst), AirDoc(ttok)) => ctx.proxy(tplst, ttok),
// Metasyntactic variables (metavariables)
(st @ PkgTpl(_), tok @ AirMeta(..)) => {
// Metavariables
(st @ (PkgTpl(_) | PkgExpr(_)), tok @ AirMeta(..)) => {
ctx.ret_or_transfer(st, tok, AirMetaAggregate::new())
}
(PkgMeta(meta), AirMeta(mtok)) => ctx.proxy(meta, mtok),
(PkgMeta(meta), AirBind(mtok)) => ctx.proxy(meta, mtok),
(PkgMeta(meta), tok @ (AirExpr(..) | AirTpl(..) | AirDoc(..))) => {
(PkgMeta(meta), AirDoc(mtok)) => ctx.proxy(meta, mtok),
(PkgMeta(meta), tok @ (AirExpr(..) | AirTpl(..))) => {
ctx.try_ret_with_lookahead(meta, tok)
}
@ -272,7 +276,7 @@ impl ParseState for AirAggregate {
// TODO: We will need to be more intelligent about this,
// since desugaring will produce metavariables in nested contexts,
// e.g. within an expression within a template.
(st @ (Pkg(..) | PkgExpr(..) | PkgOpaque(..)), AirMeta(tok)) => {
(st @ (Pkg(..) | PkgOpaque(..)), AirMeta(tok)) => {
Transition(st).err(AsgError::UnexpectedMeta(tok.span()))
}
@ -429,6 +433,42 @@ impl AirAggregate {
Root(_) => Filter,
}
}
/// Whether the active context represents an object that can be
/// lexically instantiated.
///
/// Only containers that support _instantiation_ are able to contain
/// abstract identifiers.
/// Instantiation triggers expansion,
/// which resolves metavariables and makes all abstract objects
/// therein concrete.
fn is_lexically_instantiatible(&self) -> bool {
use AirAggregate::*;
match self {
Uninit => false,
// These objects cannot be instantiated,
// and so the abstract identifiers that they own would never
// be able to be made concrete.
Root(_) => false,
Pkg(_) => false,
PkgExpr(_) => false,
// Templates are the metalinguistic abstraction and are,
// at the time of writing,
// the only containers capable of instantiation.
PkgTpl(_) => true,
// Metavariables cannot own identifiers
// (they can only reference them).
PkgMeta(_) => false,
// If an object is opaque to us then we cannot possibly look
// into it to see what needs expansion.
PkgOpaque(_) => false,
}
}
}
/// Behavior of an environment boundary when crossing environment upward
@ -823,7 +863,7 @@ impl AirAggregateCtx {
},
)?;
oi_pkg.root(&mut self.asg);
oi_pkg.root(&mut self.asg)?;
self.ooi_pkg.replace(oi_pkg);
Ok(oi_pkg)
@ -847,11 +887,33 @@ impl AirAggregateCtx {
///
/// A value of [`None`] indicates that no bindings are permitted in the
/// current context.
fn rooting_oi(&self) -> Option<ObjectIndexToTree<Ident>> {
fn rooting_oi(&self) -> Option<(&AirAggregate, ObjectIndexToTree<Ident>)> {
self.stack
.iter()
.rev()
.find_map(|st| st.active_rooting_oi())
.find_map(|st| st.active_rooting_oi().map(|oi| (st, oi)))
}
/// The active container (rooting context) for _abstract_ [`Ident`]s.
///
/// Only containers that support _instantiation_ are able to contain
/// abstract identifiers.
/// Instantiation triggers expansion,
/// which resolves metavariables and makes all abstract objects
/// therein concrete.
///
/// This utilizes [`Self::rooting_oi`] to determine the active rooting
/// context.
/// If that context does not support instantiation,
/// [`None`] is returned.
/// This method will _not_ continue looking further up the stack for a
/// context that is able to be instantiated,
/// since that would change the parent of the binding.
fn instantiable_rooting_oi(
&self,
) -> Option<(&AirAggregate, ObjectIndexToTree<Ident>)> {
self.rooting_oi()
.filter(|(st, _)| st.is_lexically_instantiatible())
}
/// The active dangling expression context for [`Expr`]s.
@ -925,17 +987,77 @@ impl AirAggregateCtx {
})
}
/// Root an identifier using the [`Self::rooting_oi`] atop of the stack.
fn defines(&mut self, name: SPair) -> Result<ObjectIndex<Ident>, AsgError> {
let oi_root = self
/// Root a concrete identifier using the [`Self::rooting_oi`] atop of
/// the stack.
///
/// This definition will index the identifier into the proper
/// environments,
/// giving it scope.
/// If the identifier is abstract,
/// it is important to use [`Self::defines_abstract`] instead so that
/// the metavariable that the identifier references will not be
/// indexed as the binding `name`.
fn defines_concrete(
&mut self,
binding_name: SPair,
) -> Result<ObjectIndex<Ident>, AsgError> {
let (_, oi_root) = self
.rooting_oi()
.ok_or(AsgError::InvalidBindContext(name))?;
.ok_or(AsgError::InvalidBindContext(binding_name))?;
Ok(self.lookup_lexical_or_missing(name).add_edge_from(
self.asg_mut(),
oi_root,
None,
))
self.lookup_lexical_or_missing(binding_name)
.defined_by(self.asg_mut(), oi_root)
}
/// Define an abstract identifier within the context of a container that
/// is able to hold dangling objects.
///
/// If the identifier is concrete,
/// then it is important to use [`Self::defines_concrete`] instead to
/// ensure that the identifier has its scope computed and indexed.
///
/// TODO: This is about to evolve;
/// document further.
fn defines_abstract(
&mut self,
meta_name: SPair,
) -> Result<ObjectIndex<Ident>, AsgError> {
match self.instantiable_rooting_oi() {
// The container cannot be instantiated and so there is no
// chance that this expression will be expanded in the future.
None => {
// Since we do not have an abstract container,
// the nearest container (if any) is presumably concrete,
// so let's reference that in the hope of making the error
// more informative.
// Note that this _does_ re-search the stack,
// but this is an error case that should seldom occur.
// If it's a problem,
// we can have `instantiable_rooting_oi` retain
// information.
let rooting_span = self
.rooting_oi()
.map(|(_, oi)| oi.widen().resolve(self.asg_ref()).span());
// Note that we _discard_ the attempted bind token
// and so remain in a dangling state.
Err(AsgError::InvalidAbstractBindContext(
meta_name,
rooting_span,
))
}
// We root a new identifier in the instantiable container,
// but we do not index it,
// since its name is not known until instantiation.
Some((_, oi_root)) => {
let oi_meta_ident = self.lookup_lexical_or_missing(meta_name);
oi_meta_ident
.new_abstract_ident(self.asg_mut(), meta_name.span())
.and_then(|oi| oi.defined_by(self.asg_mut(), oi_root))
}
}
}
/// Attempt to retrieve an identifier and its scope information from the
@ -966,6 +1088,26 @@ impl AirAggregateCtx {
.map(EnvScopeKind::into_inner)
}
/// Resolve an identifier at the scope of the provided environment and
/// retrieve its definition.
///
/// If the identifier is not in scope or does not have a definition,
/// [`None`] will be returned;
/// the caller cannot distinguish between the two using this method;
/// see [`Self::env_scope_lookup`] if that distinction is important.
#[cfg(test)]
fn env_scope_lookup_ident_dfn<O: ObjectRelatable>(
&self,
env: impl ObjectIndexRelTo<Ident>,
name: SPair,
) -> Option<ObjectIndex<O>>
where
Ident: ObjectRelTo<O>,
{
self.env_scope_lookup::<Ident>(env, name)
.and_then(|oi| oi.definition(self.asg_ref()))
}
/// Attempt to retrieve an identifier from the graph by name relative to
/// the immediate environment `imm_env`.
///
@ -1168,7 +1310,7 @@ impl<T> AsRef<T> for EnvScopeKind<T> {
}
}
impl<T, U> Functor<T, U> for EnvScopeKind<T> {
impl<T, U> Map<T, U> for EnvScopeKind<T> {
type Target = EnvScopeKind<U>;
fn map(self, f: impl FnOnce(T) -> U) -> Self::Target {

View File

@ -30,11 +30,8 @@ use super::{
AirAggregate, AirAggregateCtx,
};
use crate::{
asg::{
graph::object::{ObjectIndexRelTo, ObjectIndexTo},
ObjectKind,
},
f::Functor,
asg::{graph::object::ObjectIndexTo, ObjectKind},
f::Map,
parse::prelude::*,
};
@ -96,8 +93,12 @@ impl ParseState for AirExprAggregate {
}
(BuildingExpr(es, poi), AirExpr(ExprStart(op, span))) => {
let oi = poi.create_subexpr(ctx.asg_mut(), Expr::new(op, span));
Transition(BuildingExpr(es.push(poi), oi)).incomplete()
match poi.create_subexpr(ctx.asg_mut(), Expr::new(op, span)) {
Ok(oi) => {
Transition(BuildingExpr(es.push(poi), oi)).incomplete()
}
Err(e) => Transition(BuildingExpr(es, poi)).err(e),
}
}
(BuildingExpr(es, oi), AirExpr(ExprEnd(end))) => {
@ -123,7 +124,7 @@ impl ParseState for AirExprAggregate {
}
(BuildingExpr(es, oi), AirBind(BindIdent(id))) => {
let result = ctx.defines(id).and_then(|oi_ident| {
let result = ctx.defines_concrete(id).and_then(|oi_ident| {
oi_ident.bind_definition(ctx.asg_mut(), id, oi)
});
@ -138,20 +139,38 @@ impl ParseState for AirExprAggregate {
}
}
(BuildingExpr(es, oi), AirBind(RefIdent(name))) => {
let oi_ident = ctx.lookup_lexical_or_missing(name);
Transition(BuildingExpr(
es,
oi.ref_expr(ctx.asg_mut(), oi_ident),
))
.incomplete()
(BuildingExpr(es, oi), AirBind(BindIdentAbstract(meta_name))) => {
let result =
ctx.defines_abstract(meta_name).and_then(|oi_ident| {
oi_ident.bind_definition(ctx.asg_mut(), meta_name, oi)
});
// It is important that we do not mark this expression as
// reachable unless we successfully bind the identifier.
// Even though the identifier is abstract,
// we want to mimic the concrete structure of the graph.
match result {
Ok(oi_ident) => {
Transition(BuildingExpr(es.reachable_by(oi_ident), oi))
.incomplete()
}
Err(e) => Transition(BuildingExpr(es, oi)).err(e),
}
}
(BuildingExpr(es, oi), AirDoc(DocIndepClause(clause))) => {
oi.desc_short(ctx.asg_mut(), clause);
Transition(BuildingExpr(es, oi)).incomplete()
(BuildingExpr(es, oi), AirBind(RefIdent(name))) => {
let oi_ident = ctx.lookup_lexical_or_missing(name);
oi.ref_expr(ctx.asg_mut(), oi_ident)
.map(|_| ())
.transition(BuildingExpr(es, oi))
}
(BuildingExpr(es, oi), AirDoc(DocIndepClause(clause))) => oi
.add_desc_short(ctx.asg_mut(), clause)
.map(|_| ())
.transition(BuildingExpr(es, oi)),
(BuildingExpr(es, oi), AirDoc(DocText(text))) => Transition(
BuildingExpr(es, oi),
)
@ -188,9 +207,10 @@ impl AirExprAggregate {
oi_root: Option<ObjectIndexTo<Expr>>,
oi_expr: ObjectIndex<Expr>,
) -> Result<(), AsgError> {
oi_root
.ok_or(AsgError::DanglingExpr(oi_expr.resolve(asg).span()))?
.add_edge_to(asg, oi_expr, None);
let oi_container = oi_root
.ok_or(AsgError::DanglingExpr(oi_expr.resolve(asg).span()))?;
oi_expr.held_by(asg, oi_container)?;
Ok(())
}

View File

@ -18,18 +18,23 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use super::*;
use crate::asg::{
air::{
test::{
air_ctx_from_pkg_body_toks, air_ctx_from_toks, parse_as_pkg_body,
pkg_expect_ident_obj, pkg_expect_ident_oi, pkg_lookup,
},
Air, AirAggregate,
},
graph::object::{expr::ExprRel, Doc, ObjectRel},
ExprOp, Ident,
};
use crate::span::dummy::*;
use crate::{
asg::{
air::{
test::{
air_ctx_from_pkg_body_toks, air_ctx_from_toks,
parse_as_pkg_body, pkg_expect_ident_obj, pkg_expect_ident_oi,
pkg_lookup,
},
Air::*,
AirAggregate,
},
graph::object::{expr::ExprRel, Doc, ObjectRel},
ExprOp, Ident,
},
parse::util::spair,
};
use std::assert_matches::assert_matches;
type Sut = AirAggregate;
@ -46,13 +51,13 @@ pub fn collect_subexprs(
#[test]
fn expr_empty_ident() {
let id = SPair("foo".into(), S2);
let id = spair("foo", S2);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id),
Air::ExprEnd(S3),
let toks = [
ExprStart(ExprOp::Sum, S1),
BindIdent(id),
ExprEnd(S3),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -65,13 +70,13 @@ fn expr_empty_ident() {
#[test]
fn expr_without_pkg() {
let toks = vec![
let toks = [
// No package
// (because we're not parsing with `parse_as_pkg_body` below)
Air::ExprStart(ExprOp::Sum, S1),
ExprStart(ExprOp::Sum, S1),
// RECOVERY
Air::PkgStart(S2, SPair("/pkg".into(), S2)),
Air::PkgEnd(S3),
PkgStart(S2, spair("/pkg", S2)),
PkgEnd(S3),
];
assert_eq!(
@ -88,18 +93,18 @@ fn expr_without_pkg() {
// Note that this can't happen in e.g. NIR / TAME's source XML.
#[test]
fn close_pkg_mid_expr() {
let id = SPair("foo".into(), S4);
let id = spair("foo", S4);
#[rustfmt::skip]
let toks = vec![
Air::PkgStart(S1, SPair("/pkg".into(), S1)),
Air::ExprStart(ExprOp::Sum, S2),
Air::PkgEnd(S3),
let toks = [
PkgStart(S1, spair("/pkg", S1)),
ExprStart(ExprOp::Sum, S2),
PkgEnd(S3),
// RECOVERY: Let's finish the expression first...
Air::BindIdent(id),
Air::ExprEnd(S5),
BindIdent(id),
ExprEnd(S5),
// ...and then try to close again.
Air::PkgEnd(S6),
PkgEnd(S6),
];
assert_eq!(
@ -123,20 +128,20 @@ fn close_pkg_mid_expr() {
#[test]
fn open_pkg_mid_expr() {
let pkg_a = SPair("/pkg".into(), S1);
let pkg_nested = SPair("/pkg-nested".into(), S3);
let id = SPair("foo".into(), S4);
let pkg_a = spair("/pkg", S1);
let pkg_nested = spair("/pkg-nested", S3);
let id = spair("foo", S4);
#[rustfmt::skip]
let toks = vec![
Air::PkgStart(S1, pkg_a),
Air::ExprStart(ExprOp::Sum, S2),
Air::PkgStart(S3, pkg_nested),
let toks = [
PkgStart(S1, pkg_a),
ExprStart(ExprOp::Sum, S2),
PkgStart(S3, pkg_nested),
// RECOVERY: We should still be able to complete successfully.
Air::BindIdent(id),
Air::ExprEnd(S5),
BindIdent(id),
ExprEnd(S5),
// Closes the _original_ package.
Air::PkgEnd(S6),
PkgEnd(S6),
];
assert_eq!(
@ -163,23 +168,23 @@ fn open_pkg_mid_expr() {
#[test]
fn expr_non_empty_ident_root() {
let id_a = SPair("foo".into(), S2);
let id_b = SPair("bar".into(), S2);
let id_a = spair("foo", S2);
let id_b = spair("bar", S2);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// Identifier while still empty...
Air::BindIdent(id_a),
BindIdent(id_a),
Air::ExprStart(ExprOp::Sum, S3),
ExprStart(ExprOp::Sum, S3),
// (note that the inner expression _does not_ have an ident
// binding)
Air::ExprEnd(S4),
ExprEnd(S4),
// ...and an identifier non-empty.
Air::BindIdent(id_b),
Air::ExprEnd(S6),
BindIdent(id_b),
ExprEnd(S6),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -197,20 +202,20 @@ fn expr_non_empty_ident_root() {
// which only becomes reachable at the end.
#[test]
fn expr_non_empty_bind_only_after() {
let id = SPair("foo".into(), S2);
let id = spair("foo", S2);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// Expression root is still dangling at this point.
Air::ExprStart(ExprOp::Sum, S2),
Air::ExprEnd(S3),
ExprStart(ExprOp::Sum, S2),
ExprEnd(S3),
// We only bind an identifier _after_ we've created the expression,
// which should cause the still-dangling root to become
// reachable.
Air::BindIdent(id),
Air::ExprEnd(S5),
BindIdent(id),
ExprEnd(S5),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -225,11 +230,11 @@ fn expr_non_empty_bind_only_after() {
// since they're either mistakes or misconceptions.
#[test]
fn expr_dangling_no_subexpr() {
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// No `BindIdent`,
// so this expression is dangling.
Air::ExprEnd(S2),
ExprEnd(S2),
];
// The error span should encompass the entire expression.
@ -251,14 +256,14 @@ fn expr_dangling_no_subexpr() {
#[test]
fn expr_dangling_with_subexpr() {
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// Expression root is still dangling at this point.
Air::ExprStart(ExprOp::Sum, S2),
Air::ExprEnd(S3),
ExprStart(ExprOp::Sum, S2),
ExprEnd(S3),
// Still no ident binding,
// so root should still be dangling.
Air::ExprEnd(S4),
ExprEnd(S4),
];
let full_span = S1.merge(S4).unwrap();
@ -280,23 +285,23 @@ fn expr_dangling_with_subexpr() {
#[test]
fn expr_dangling_with_subexpr_ident() {
let id = SPair("foo".into(), S3);
let id = spair("foo", S3);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// Expression root is still dangling at this point.
Air::ExprStart(ExprOp::Sum, S2),
ExprStart(ExprOp::Sum, S2),
// The _inner_ expression receives an identifier,
// but that should have no impact on the dangling status of
// the root,
// especially given that subexpressions are always reachable
// anyway.
Air::BindIdent(id),
Air::ExprEnd(S4),
BindIdent(id),
ExprEnd(S4),
// But the root still has no ident binding,
// and so should still be dangling.
Air::ExprEnd(S5),
ExprEnd(S5),
];
let full_span = S1.merge(S5).unwrap();
@ -323,18 +328,18 @@ fn expr_dangling_with_subexpr_ident() {
// but this also protects against potential future breakages.
#[test]
fn expr_reachable_subsequent_dangling() {
let id = SPair("foo".into(), S2);
let id = spair("foo", S2);
#[rustfmt::skip]
let toks = vec![
let toks = [
// Reachable
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id),
Air::ExprEnd(S3),
ExprStart(ExprOp::Sum, S1),
BindIdent(id),
ExprEnd(S3),
// Dangling
Air::ExprStart(ExprOp::Sum, S4),
Air::ExprEnd(S5),
ExprStart(ExprOp::Sum, S4),
ExprEnd(S5),
];
// The error span should encompass the entire expression.
@ -362,18 +367,18 @@ fn expr_reachable_subsequent_dangling() {
// Recovery from dangling expression.
#[test]
fn recovery_expr_reachable_after_dangling() {
let id = SPair("foo".into(), S4);
let id = spair("foo", S4);
#[rustfmt::skip]
let toks = vec![
let toks = [
// Dangling
Air::ExprStart(ExprOp::Sum, S1),
Air::ExprEnd(S2),
ExprStart(ExprOp::Sum, S1),
ExprEnd(S2),
// Reachable, after error from dangling.
Air::ExprStart(ExprOp::Sum, S3),
Air::BindIdent(id),
Air::ExprEnd(S5),
ExprStart(ExprOp::Sum, S3),
BindIdent(id),
ExprEnd(S5),
];
// The error span should encompass the entire expression.
@ -415,21 +420,21 @@ fn recovery_expr_reachable_after_dangling() {
#[test]
fn expr_close_unbalanced() {
let id = SPair("foo".into(), S3);
let id = spair("foo", S3);
#[rustfmt::skip]
let toks = vec![
let toks = [
// Close before _any_ open.
Air::ExprEnd(S1),
ExprEnd(S1),
// Should recover,
// allowing for a normal expr.
Air::ExprStart(ExprOp::Sum, S2),
Air::BindIdent(id),
Air::ExprEnd(S4),
ExprStart(ExprOp::Sum, S2),
BindIdent(id),
ExprEnd(S4),
// And now an extra close _after_ a valid expr.
Air::ExprEnd(S5),
ExprEnd(S5),
];
let mut sut = parse_as_pkg_body(toks);
@ -467,26 +472,26 @@ fn expr_close_unbalanced() {
// for non-associative expressions.
#[test]
fn sibling_subexprs_have_ordered_edges_to_parent() {
let id_root = SPair("root".into(), S1);
let id_root = spair("root", S1);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
let toks = [
ExprStart(ExprOp::Sum, S1),
// Identify the root so that it is not dangling.
Air::BindIdent(id_root),
BindIdent(id_root),
// Sibling A
Air::ExprStart(ExprOp::Sum, S3),
Air::ExprEnd(S4),
ExprStart(ExprOp::Sum, S3),
ExprEnd(S4),
// Sibling B
Air::ExprStart(ExprOp::Sum, S5),
Air::ExprEnd(S6),
ExprStart(ExprOp::Sum, S5),
ExprEnd(S6),
// Sibling C
Air::ExprStart(ExprOp::Sum, S7),
Air::ExprEnd(S8),
Air::ExprEnd(S9),
ExprStart(ExprOp::Sum, S7),
ExprEnd(S8),
ExprEnd(S9),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -516,21 +521,21 @@ fn sibling_subexprs_have_ordered_edges_to_parent() {
#[test]
fn nested_subexprs_related_to_relative_parent() {
let id_root = SPair("root".into(), S1);
let id_suba = SPair("suba".into(), S2);
let id_root = spair("root", S1);
let id_suba = spair("suba", S2);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1), // 0
Air::BindIdent(id_root),
let toks = [
ExprStart(ExprOp::Sum, S1), // 0
BindIdent(id_root),
Air::ExprStart(ExprOp::Sum, S2), // 1
Air::BindIdent(id_suba),
ExprStart(ExprOp::Sum, S2), // 1
BindIdent(id_suba),
Air::ExprStart(ExprOp::Sum, S3), // 2
Air::ExprEnd(S4),
Air::ExprEnd(S5),
Air::ExprEnd(S6),
ExprStart(ExprOp::Sum, S3), // 2
ExprEnd(S4),
ExprEnd(S5),
ExprEnd(S6),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -556,18 +561,18 @@ fn nested_subexprs_related_to_relative_parent() {
fn expr_redefine_ident() {
// Same identifier but with different spans
// (which would be the case in the real world).
let id_first = SPair("foo".into(), S2);
let id_dup = SPair("foo".into(), S3);
let id_first = spair("foo", S2);
let id_dup = spair("foo", S3);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id_first),
let toks = [
ExprStart(ExprOp::Sum, S1),
BindIdent(id_first),
Air::ExprStart(ExprOp::Sum, S3),
Air::BindIdent(id_dup),
Air::ExprEnd(S4),
Air::ExprEnd(S5),
ExprStart(ExprOp::Sum, S3),
BindIdent(id_dup),
ExprEnd(S4),
ExprEnd(S5),
];
let mut sut = parse_as_pkg_body(toks);
@ -606,32 +611,32 @@ fn expr_redefine_ident() {
fn expr_still_dangling_on_redefine() {
// Same identifier but with different spans
// (which would be the case in the real world).
let id_first = SPair("foo".into(), S2);
let id_dup = SPair("foo".into(), S5);
let id_dup2 = SPair("foo".into(), S8);
let id_second = SPair("bar".into(), S9);
let id_first = spair("foo", S2);
let id_dup = spair("foo", S5);
let id_dup2 = spair("foo", S8);
let id_second = spair("bar", S9);
#[rustfmt::skip]
let toks = vec![
let toks = [
// First expr (OK)
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id_first),
Air::ExprEnd(S3),
ExprStart(ExprOp::Sum, S1),
BindIdent(id_first),
ExprEnd(S3),
// Second expr should still dangle due to use of duplicate
// identifier
Air::ExprStart(ExprOp::Sum, S4),
Air::BindIdent(id_dup),
Air::ExprEnd(S6),
ExprStart(ExprOp::Sum, S4),
BindIdent(id_dup),
ExprEnd(S6),
// Third expr will error on redefine but then be successful.
// This probably won't happen in practice with TAME's original
// source language,
// but could happen at e.g. a REPL.
Air::ExprStart(ExprOp::Sum, S7),
Air::BindIdent(id_dup2), // fail
Air::BindIdent(id_second), // succeed
Air::ExprEnd(S10),
ExprStart(ExprOp::Sum, S7),
BindIdent(id_dup2), // fail
BindIdent(id_second), // succeed
ExprEnd(S10),
];
let mut sut = parse_as_pkg_body(toks);
@ -691,27 +696,27 @@ fn expr_still_dangling_on_redefine() {
#[test]
fn expr_ref_to_ident() {
let id_foo = SPair("foo".into(), S2);
let id_bar = SPair("bar".into(), S6);
let id_foo = spair("foo", S2);
let id_bar = spair("bar", S6);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id_foo),
let toks = [
ExprStart(ExprOp::Sum, S1),
BindIdent(id_foo),
// Reference to an as-of-yet-undefined id (okay),
// with a different span than `id_bar`.
Air::RefIdent(SPair("bar".into(), S3)),
Air::ExprEnd(S4),
RefIdent(spair("bar", S3)),
ExprEnd(S4),
//
// Another expression to reference the first
// (we don't handle cyclic references until a topological sort,
// so no point in referencing ourselves;
// it'd work just fine here.)
Air::ExprStart(ExprOp::Sum, S5),
Air::BindIdent(id_bar),
Air::ExprEnd(S7),
ExprStart(ExprOp::Sum, S5),
BindIdent(id_bar),
ExprEnd(S7),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -748,23 +753,23 @@ fn expr_ref_to_ident() {
#[test]
fn idents_share_defining_pkg() {
let id_foo = SPair("foo".into(), S3);
let id_bar = SPair("bar".into(), S5);
let id_baz = SPair("baz".into(), S6);
let id_foo = spair("foo", S3);
let id_bar = spair("bar", S5);
let id_baz = spair("baz", S6);
// An expression nested within another.
#[rustfmt::skip]
let toks = vec![
Air::PkgStart(S1, SPair("/pkg".into(), S1)),
Air::ExprStart(ExprOp::Sum, S2),
Air::BindIdent(id_foo),
let toks = [
PkgStart(S1, spair("/pkg", S1)),
ExprStart(ExprOp::Sum, S2),
BindIdent(id_foo),
Air::ExprStart(ExprOp::Sum, S4),
Air::BindIdent(id_bar),
Air::RefIdent(id_baz),
Air::ExprEnd(S7),
Air::ExprEnd(S8),
Air::PkgEnd(S9),
ExprStart(ExprOp::Sum, S4),
BindIdent(id_bar),
RefIdent(id_baz),
ExprEnd(S7),
ExprEnd(S8),
PkgEnd(S9),
];
let ctx = air_ctx_from_toks(toks);
@ -789,15 +794,15 @@ fn idents_share_defining_pkg() {
#[test]
fn expr_doc_short_desc() {
let id_expr = SPair("foo".into(), S2);
let clause = SPair("short desc".into(), S3);
let id_expr = spair("foo", S2);
let clause = spair("short desc", S3);
#[rustfmt::skip]
let toks = vec![
Air::ExprStart(ExprOp::Sum, S1),
Air::BindIdent(id_expr),
Air::DocIndepClause(clause),
Air::ExprEnd(S4),
let toks = [
ExprStart(ExprOp::Sum, S1),
BindIdent(id_expr),
DocIndepClause(clause),
ExprEnd(S4),
];
let ctx = air_ctx_from_pkg_body_toks(toks);
@ -813,3 +818,73 @@ fn expr_doc_short_desc() {
oi_docs.collect::<Vec<_>>(),
);
}
// Binding an abstract identifier to an expression means that the expression
// may _eventually_ be reachable after expansion,
// but it is not yet.