And everything else.
This is a big (important) change; it addresses one of the greatest
pains of the system.
Keeps were added during the DSL rewrite (to support symbols and such)
to work around the issue that there was no symbol-driven map; it
allowed symbols to persist disjoint from the `__yield' dependency
graph so that they could be mapped back and used by external systems.
The problem with that is that it's both messy (coupling the concept of
external dependencies with the actual code) and difficult to work
with. It had a huge performance impact on the linker for two reasons:
- Checking whether a package had already been seen and importing the
keeps on first visit was expensive because of tree searching and
manipulation; and
- _every_ keep was imported and processed by the linker, even if it
wouldn't end up being used by a particular program.
The later especially had huge performance impacts on the entire
system.
The entire dependency graph is now map-driven, with the exception of
the implicit `__yield' (which will eventually be moved into the map as
well and the magic `lv:yield' removed in favor of a template).
Performance-wise: our largest program ("dwelling") has many thousands
of symbols and the largest package imported the majority of them, many
of them unneeded, as the result of @keep subgraphs. Compilation of
the largest package within that (for the UI) took about a minute and a
half and ate up ~6GiB of RAM, for what really is a trivial task of
resolving externs, some basic symbol processing, a topological sort,
and ordering code fragments.
After this change, it takes ~15s and less than 2GiB of RAM. Still a
lot---and more improvements can be made---but much, much better.
@keep and friends was left in rater.xsd so that nothing breaks while
code is cleaned up; it'll be removed in the future.
* src/current/compiler/linker.xsl: Remove @keep support.
* src/current/dot/attr-keep.xsl: Remove now-unneeded template.
* src/current/dot/defnode.xsl: Remove @keep and related.
* src/current/include/preproc/eligclass.xsl: Remove @keep and related.
* src/current/include/preproc/expand.xsl: Remove @keep and related.
* src/current/include/preproc/macros.xsl: Remove @keep and related.
* src/current/include/preproc/symtable.xsl: Remove @keep and related.
* src/current/rater.xsd: Add TODO to remove @keep and friends.
This solves a long-standing problem whereby relative paths are not
properly resolved, leading to incorrect symbol conflicts when relative
paths to the same package vary between two imports.
See doc/notes/path-processing for an illustration.
* src/current/include/preproc/path.xsl
(preproc:resolve-relative-import): Add function.
(preproc:resolv-path, preproc:resolve-path, preproc:get-path): Add
functions to invoke existing templates.
* src/current/include/preproc/symtable.xsl (preproc:symimport): Use
`preproc:resolve-relative-import'.
* doc/notes/path-processing: Add notes.
This is a backwards-incompatible change that, like the input map,
requires the use of symbols in the return map. This will allow us to
forego the use of @keep and will have the return map be the authority
of what gets linked (all of its dependencies).
* src/current/compiler/map.xsl: Add symbol support to return-map.
This allows for the proper importing of symbols into the package
generated by the map compiler, which in turn allows for processing
their default values.
`set_defaults' wasn't in scope of maps.
* src/current/compiler/js.xsl (compiler:exit-rater lv:package):
Remove static output.
* src/current/compiler/linker.xsl (l:link-deps lv:package):
Link static after all other blocks, at highest scope within the
compiled module.
The `preproc:param-dim' template must remain for now, as it is used by
others.
* src/current/include/preproc/symtable.xsl
(preproc:symtable)[lv:param]: Extracted template.
* src/symtable/symbols.xsl
(preproc:symtable)[lv:param]: Added template.
* test/symtable/symbols.xsl
(lv:param): Scenario added.
It's important that others understand the system before I keep adding to
the mystery. This is at least a good start.
It also obviates certain awkward design issues that have evolved over
time and need addressing. Note that it also mentions that `keep' is
marked for removal---this is the situation that prompted this
documentation; changes will be made to work toward its removal to
improve the terrible linker performance when given many thousands of
symbols defined in over 500 separate packages.
* src/symtable.xsl: Add menu for symbols.
* src/symtable/symbols.xsl: Added.
This allows for nice formatting with newlines (to avoid excessively long
lines) without affecting the output.
* src/current/include/preproc/expand.xsl
(preproc:expand lv:classify/@desc): Normalize lv:classify/@desc whitespace
(Copyright headers will be added in the next commit; these are the
original files, unaltered in any way.)
The internal project name at LoVullo is simply "Calc DSL". This
liberates the entire thing. If anything was missed, I'll be added
later.
To continue building at LoVullo with this move, symlinks are used for
the transition; this is the exact code that is used in production.
There is a lot here---over 25,000 lines. Much of it is in disarray from
the environment surrounding its development, but it does work well for
what it was intended to do.
(LoVullo folks: fork point is 65723a0 in calcdsl.git.)
The thrush reference was also removed, as TAME does not currently make
use of it.
* .gitignore: Update path accordingly.
* configure.ac: Update path accordingly.
* src/init.xsl.in: Moved from conf.xsl.in.
This makes changes necessary to new ref format (which includes arity).
* src/graph.xsl: @ref function arity added
* src/preproc/expand/expand-sequence.xsl: @ref function arity added
* tools/literate-xsl: Submodule updated
This was already happening, but formally test and document it, since
it's a very useful property.
* src/graph.xsl (graph:union): Document dedupe behavior
* test/graph-test.xsl: Add test data
* test/graph.xspec: Add dedupe test for graph:union
Mike Gerwitz
Mike Gerwitz