We need to cut down on symbol imports as much as possible; the whole system
starts dragging if we are importing thousands of symbols into a single
package.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate,c:*]: Mark
as local if `@preproc:generated`.
* src/current/include/preproc/template.xsl (preproc:macros)[lv:inline-template]:
Mark generated templates as such.
* src/symtable/symbols.xsl (preproc:symtable)[lv:template]: Mark as local if
`@preproc:generated'.
* src/current/compiler/js.xsl (compile-class-condtion)[lv:rate]: Do not
consider @no's in predicate generation when `@preproc:gentle-no' is set.
* src/current/include/preproc/macros.xsl (preproc:macros)[lv:rate-each]: Set
`@preproc:gentle-no' on generated `lv:rate', since the generator handles
`@no' itself.
* src/current/include/preproc/template.xsl
(preproc:gen-param-value)[lv:param-sym-value]: Suppress warning for
missing symbol and yield empty string if `@ignore-missing='true'`.
The problem with this implementation was that, any time a generator had an
associated generated @yields (which is common), it wouldn't be included in
the summary page.
We can address this in the future. It's not necessarily that it was
incorrect; it's just how the system made use of it.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate]:
Do not mark @preproc:yields-generated symbols as @preproc:generated.
any/all blocks within classifications are extracted into their own
classifications. When they have sibling nodes, the extracted
classifications did not have their templates expanded. Ouch.
This is a situation that should never happen (I haven't yet determined
whether or not we should support this type of thing in the future), but when
it does, do something intelligent.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[c:*[@generates]]:
Consider only first ancestor lv:rate.
For example, with template-generating templates, if a reference is unknown
and therefore determined to be blank, an attribute might be completely
removed while the template is being generated. (See the /when package in
core for an example.) That is not good.
* src/current/include/preproc/template
(preproc:expand-template): Add `tpl' tunneling param to
preproc:apply-template application.
(preproc:apply-template)[@*]: Retain param reference if applying template
does not define it.
This is important for template-generating templates.
* src/current/include/preproc/template.xsl (lv:if,lv:unlesS):
Retain conditional nodes verbatim when referenced param is unknown.
* src/current/include/preproc/expand.xsl
(lv:classify): Set `@preproc:yields-generated' if `@yields' was not
previously defined.
* src/current/include/preproc/macros.xsl:
(lv:rate-each): Set `@preproc:yields-generated` if `@yields' was not
previously defined.
* src/current/include/preproc/symtable.xsl:
(lv:rate, lv:classify): Set `@preproc:generated` on symbol if
`@preproc:yields-generated'.
* src/current/include/preproc/template.xsl
(preproc:gen-param-value)[lv:param-add]: Resolve value of @value before
processing numeric expression (to support template variables). Error
when any values are not numeric rather than producing NaN.
This is imperfect, but will work based on current assumptions.
* src/current/include/preproc/expand.xsl (preproc:expand)[c:const,lv:const):
If '@' is found, do not perform expansion.
When a symbol is imported as local, and subsequently directly imported using
another package with @export, the symbol needs to be overridden as
non-local.
* src/current/include/preproc/symtable.xsl
(preproc:sym-discover): Deduplicate based on local status.
(preproc:symtable-process-symbols): Retain non-local symbols when only
local version of the symbol exists.
If it is copied an yields an empty value, then it is later converted into a
0, which is wholly incorrect in the case of generators.
* src/current/include/preproc/macros.xsl (preproc:macros)[lv:rate-each]:
Conditionally copy `@dim'.
This optimization (which already existed) is supposed to remove
c:sum/c:product nodes if they contain one or fewer calculations, but it
failed to remove them when inline templates were used (because it creates,
well, a template inline).
Instead, we'll assume that any lv:* nodes handled by the preprocessor will
be okay. In the case of inline templates, they receive a preproc attribute
identifying what created it.
I hope this is a valid assumption. This will be thoroughly tested. If only
we had test cases for the preprocessor.
* src/current/include/preproc/expand.xsl: (preproc:expand)[c:sum,c:product]:
Consider optimization for `lv:*[@preproc:*]'.
This was recently introduced and was noticed because c:let/@name was not
being generated under certain circumstances.
* src/current/include/preproc/expand.xsl (preproc:expand): Continue applying
templates after @dim expansion (rather than copying child nodes).
This (hopefully) prevents metadata from leaking outside of template
expansions where they are not wanted.
* src/current/include/preproc/macros.xsl (preproc:macropass): Strip
cruft (new barrier nodes) on final pass.
* src/current/include/preproc/template.xsl (preproc:expand-template):
Enclose template expansion in preproc:tpl-barrier (which is stripped on
the final pass).
(preproc:apply-template)[lv:param-meta]: Document template. "Hoist"
node (set attribute) if not contained within lv:param-copy.
(preproc:gen-param-value)[lv:param-inherit]: Consider hoisted nodes behind
barrier.
This allows for debugging expansions, which are otherwise confusing and
difficult. This will be valuable information for the Summary Pages as
well (see future commit).
* src/current/include/preproc/template.xsl (preproc:expand-template): Mark
toplevel nodes after expansion.
(preproc:mark-tpl-expansion): Add templates.
Other code relied on expand-sequence, but this really should be implicit,
since it is deceptively useless otherwise.
* src/current/include/preproc/template.xsl (preproc:macros): Defer
processing of inline-template with sym-set until symbol table is
available.
This allows the result of a rate block to be a matrix; there was previously
no way for a named value to be assigned a matrix unless it was a parameter.
This is a bit of a kluge---the compiler won't discover the proper type
information and won't perform the proper safeguards.
* src/current/calc.xsd (sum)[@dim]: Add attribute.
* src/current/compiler/js-calc.xsl: Add xs namespace.
(compile-calc): Do not perform casting when @dim > 1.
* src/current/include/preproc/expand.xsl
(lv:rate-each): Include @dim in c:sum expansion.
* src/current/include/preproc/macros.xsl:
(c:*/@generates): Use @dim to determine symbol dimensions.
* src/current/include/preproc/expand.xsl: Parse @dim aliases (e.g. "vector",
"matrix").
A classification used to be generated for each param for convenience;
this has been deprecated by the progui package generation (when `q:'
is used).
* src/current/include/preproc/macros.xsl: Remove template.
And everything else.
This is a big (important) change; it addresses one of the greatest
pains of the system.
Keeps were added during the DSL rewrite (to support symbols and such)
to work around the issue that there was no symbol-driven map; it
allowed symbols to persist disjoint from the `__yield' dependency
graph so that they could be mapped back and used by external systems.
The problem with that is that it's both messy (coupling the concept of
external dependencies with the actual code) and difficult to work
with. It had a huge performance impact on the linker for two reasons:
- Checking whether a package had already been seen and importing the
keeps on first visit was expensive because of tree searching and
manipulation; and
- _every_ keep was imported and processed by the linker, even if it
wouldn't end up being used by a particular program.
The later especially had huge performance impacts on the entire
system.
The entire dependency graph is now map-driven, with the exception of
the implicit `__yield' (which will eventually be moved into the map as
well and the magic `lv:yield' removed in favor of a template).
Performance-wise: our largest program ("dwelling") has many thousands
of symbols and the largest package imported the majority of them, many
of them unneeded, as the result of @keep subgraphs. Compilation of
the largest package within that (for the UI) took about a minute and a
half and ate up ~6GiB of RAM, for what really is a trivial task of
resolving externs, some basic symbol processing, a topological sort,
and ordering code fragments.
After this change, it takes ~15s and less than 2GiB of RAM. Still a
lot---and more improvements can be made---but much, much better.
@keep and friends was left in rater.xsd so that nothing breaks while
code is cleaned up; it'll be removed in the future.
* src/current/compiler/linker.xsl: Remove @keep support.
* src/current/dot/attr-keep.xsl: Remove now-unneeded template.
* src/current/dot/defnode.xsl: Remove @keep and related.
* src/current/include/preproc/eligclass.xsl: Remove @keep and related.
* src/current/include/preproc/expand.xsl: Remove @keep and related.
* src/current/include/preproc/macros.xsl: Remove @keep and related.
* src/current/include/preproc/symtable.xsl: Remove @keep and related.
* src/current/rater.xsd: Add TODO to remove @keep and friends.
This solves a long-standing problem whereby relative paths are not
properly resolved, leading to incorrect symbol conflicts when relative
paths to the same package vary between two imports.
See doc/notes/path-processing for an illustration.
* src/current/include/preproc/path.xsl
(preproc:resolve-relative-import): Add function.
(preproc:resolv-path, preproc:resolve-path, preproc:get-path): Add
functions to invoke existing templates.
* src/current/include/preproc/symtable.xsl (preproc:symimport): Use
`preproc:resolve-relative-import'.
* doc/notes/path-processing: Add notes.
The `preproc:param-dim' template must remain for now, as it is used by
others.
* src/current/include/preproc/symtable.xsl
(preproc:symtable)[lv:param]: Extracted template.
* src/symtable/symbols.xsl
(preproc:symtable)[lv:param]: Added template.
* test/symtable/symbols.xsl
(lv:param): Scenario added.
This allows for nice formatting with newlines (to avoid excessively long
lines) without affecting the output.
* src/current/include/preproc/expand.xsl
(preproc:expand lv:classify/@desc): Normalize lv:classify/@desc whitespace
(Copyright headers will be added in the next commit; these are the
original files, unaltered in any way.)
The internal project name at LoVullo is simply "Calc DSL". This
liberates the entire thing. If anything was missed, I'll be added
later.
To continue building at LoVullo with this move, symlinks are used for
the transition; this is the exact code that is used in production.
There is a lot here---over 25,000 lines. Much of it is in disarray from
the environment surrounding its development, but it does work well for
what it was intended to do.
(LoVullo folks: fork point is 65723a0 in calcdsl.git.)
Mike Gerwitz
Mike Gerwitz