This is a first step (low-hanging-fruit kinda thing) for improving the
performance of symbol resolution, where the compiler has to figure out the
dimensions of a symbol by first resolving its dependencies,
recursively. This is approximately an O(n³) polynomial-time algorithm _per
recursive step_. Yikes.
This is traditionally where dynamic programming methods would be used, but
that's considerably more difficult in a immutable languages like XSLT, so
I'll do my best without. (Saxon does offer some support for mutability, but
I'd prefer to avoid it if possible.)
This first change improves performance 30--40%. For example, on two large
packages we have, build times drop from 55s to 35s and from 1m42s to 1m13s
respectively.
Good start, but much more to be done!
* src/current/include/preproc/package.xsl (preproc:resolv-syms)[lv:package]:
Compute maps for preproc:symtable and preproc:sym-deps at each recursive
step. Pass along via tunneling.
(preproc:resolv-syms)[preproc:sym]: Use them.
DEV-4354
This only saves 1--2s on a 30s run, but I want to move into this direction,
so it'll simplify future refactoring if I just add it. Small changes like
these will accumulate, too.
* src/current/compiler/linker.xsl (l:orig-package, l:root-symtable-map): New
variables.
(l:resov-extern): Use it.
A bunch of failing pipelines apparently wasn't obvious to me. And shame on
me for not running these locally; I forgot that the part of the system that
I touched had tests.
This was broken by b6cfdb4221.
This now uses year ranges, which I'll update annually.
This also renames "R-T Specialty" to "Ryan Specialty Group". The latter is
the parent company of the former. I was originally employed under the
former when LoVullo Associates was purchased, by I now work for the parent
company.
The previous commit made dependency lists optional for certain symbols. The
Summary Page needs to be updated to permit such a thing.
The whole Summary Page needs aggressive refactoring, though, so this doesn't
bother checking for `no-deps' to see if this is a bad thing.
* src/current/summary.xsl (typeset-final)[preproc:sym-ref]: Permit missing
symbol dependencies.
(lv:param|lv:const|lv:item): Likewise.
This is a significant performance improvement for dependency
generation (which is responsible for building the dependency graph for a
package).
The previous algorithm ran in O(n²) time: it would iterate over the given
symbol table, and for _each_ symbol, do a linear scan of the entire document
to search for the corresponding source block. This resulted in explosive
depgen time for larger packages.
This makes the algorithm run in O(n) by:
- Using an XSLT 3 map for the symbol table for O(1) lookups; and
- Iterating over the _document_ a single time rather than the symbol
table, referencing the symbol table as needed (in O(1) time).
There are other parts of the system that can benefit from these same
improvements. This is important, since we need to be able to handle many
thousands of symbols efficiently.
* src/current/compiler/linker.xsl (l:depgen-sym): Recognize smybol `no-deps'
property, permitting missing dependencies. This allows us to avoid
creating nonsense nodes just to satisfy the linker, while still allowing
the linker to perform essential checks to defend against compiler bugs.
* src/current/compiler/map.xsl (lvmc:stub-symtable): Set @no-deps on
`___head' and `___tail' symbols.
(lvmc:mapsym): Set `no-deps' as appropriate on map symbols.
(preproc:depgen)[lvm:map[@from]]: Generate `preproc:sym-dep' node, which
is now expected by the depgen process.
(preproc:depgen)[lvm:map[*]]: Likewise.
(preproc:depgen)[*[@lvmc:type='retmap']//lvmm:map[@from]]: Remove
unnecessary template.
(preproc:symtable)[lvm:map[@value]]: Pass `no-deps' to `lvmc:mapsym'.
* src/current/include/depgen.xsl (preproc:depgen)[preproc:symtable]: Create
and use XSLT 3 map in place of `preproc:symtable' tree. This allows for
constant-time lookups. Provide to templates via tunnelling. Use it in
place of exiting tree references. Process source tree rather than
iterating over symbol table.
(preproc:depgen)[lv:rate, c:sum[@generates], c:product[@generates],
lv:classify, lv:function/lv:param, lv:function, lv:typedef]: Produce
`preproc:sym-dep' nodes (which was previously done while iterating
over the symbol table).
(preproc:depgen)[preproc:sym]: Remove all such processing, since we no
longer iterate over the symbol table.
(preproc:depgen)[c:value-of]: Use symtable map.
(preproc:depgen-match): Likewise.
(preproc:depgen)[lv:union]: Modify to handle changes to lv:typedef
template.
(preproc:depgen)[text()]: Remove and replace with `node()'.
* src/current/include/preproc/package.xsl (preproc:resolv-syms): Remove
logging of symbol resolution. This has a slight performace impact since
there is a lot of output.
* src/current/include/preproc/symtable.xsl
(lv:function/lv:param, c:let/c:Values/c:value): Set `no-deps'.
* src/symtable/symbols.xsl: Add documentation of `no-deps'.
(preproc:symtable)[lv:meta]: Set `no-deps'.
These provide a more pleasent abstraction than having to reference CMP_OP_*
constants.
* core/test/core/vector/interpolate.xml: {t:when=>t:where-eq}.
* core/test/core/vector/table.xml: Likewise, but using the other variants
where appropriate given the value of `@op'.
* core/vector/interpolate.xml: Likewise.
* core/vector/table.xml (_when_, _where_): Rename former to latter and
provide deprecation warning.
(_when-lt_, _when-lte_, _when-gt_, _when-gte_): Add abstractions.
* src/current/rater.xsd: Permit template variable as tenplate name.
This is fairly primitive support and it completely sidesteps the bisect
algorithm for now. The next commit will abstract this a little bit further
to make it less awkward to use.
* core/test/core/vector/table.xml: New test cases.
* core/vector/filter.xml (CmpOp): New typedef.
(mfilter): Document that bisecting will not happen unless `CMP_OP_EQ'
is used. Implement that restriction.
[op]: New parameter. Provide it to `mrange'.
(_mfilter, _mrange_cmp): Rename from `_mfilter'. Implement new comparison
check based on `op'
[op]: New argument.
* core/vector/table.xml (_when_)[@op@]: New param. Add it to the produced
vector.
(_mquery): Unpack op (from `_when_') in call to `mfilter'.
Just trying to clean up a little as I go to start to make it easier
to understand.
* core/vector/filter.xml: Use _when-*_ templates and c:recurse.
* core/vector/table.xml: Likewise.
It's going to be like TeX before you know it... ._.
* src/current/include/preproc/package.xsl (preproc:tpl-check)
[lv:template|lv:const|lv:typedef|lv:param-copy]: Add lv:param-copy.
* src/current/include/preproc/template.xsl (preproc:apply-template)
[lv:expand-barrier, lv:skip-child-expansion]: New expansion control
structures.
This is a much more useful description if present.
* src/current/include/preproc/macros.xsl (preproc:macros)[c:value-of...]:
Default generated constant description to @label.
The term "set" is all wrong---it is actally intended to be a vector, and can
absolutely have duplicate elements (and often does).
* src/current/calc.xsd (vector): Add, recommending in place of `set'.
* src/current/compiler/js-calc.xsl (compile-calc)[c:set|c:vector]:
Add `c:vector' and provide deprecation notice for `c:set'.
* src/current/include/calc-display.xsl (c:set|c:vector): Likewise.
* core/test/spec.xml (_describe_): Enclose aggregate classification in a
series of nested expand-sequence to work around bug (described in
comment), which was causing test cases to not be compiled.
A better option is to pre-process all inputs, but I need a quick
fix to my stupidity. 0||""==="".
* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Correct oval default.
I wanted to get this section started so that I can easily add to it when I
have small bits of time to do so. Our documentation needs to improve.
* doc/Makefile.am (tame_TEXINFOS): Add `concept.texi'.
* doc/concept.texi: New file.
* doc/preproc.texi: Remove accidentally added input line.
* doc/tame.texi (menu): Add `Core Concepts' node.
I need to revert this for now because it breaks YAML test cases. The proper
fix is a more expressive type system with dependent types that would allow
it to know the proper number of indexes to initialize relative to other
inputs. I wanted to implement this anyway to help catch iteration-related
bugs.
I'm tabling this for now, though, since I have other things that I need to
work on.
This reverts commit 4406cbe553.
This includes, notably, the Developer Notes feature. I did not copy any
SRCUI stuff since this project uses literate documentation, but I'll add it
if it seems like it will be useful. Barely any of the project is written
literately right now.
* .gitignore: `{=>/}config.*'.
* configure.ac (SET_DEVNOTES): New variable.
(AC_CONFIG_FILES): Add `doc/config.texi'.
* doc/.gitignore (config.texi): Ignore (generated).
* doc/Makefile.am (tame_TEXINFOS): Add `macros.texi' and `config.texi'.
* doc/config.texi.in: New file.
* doc/macros.texi: New file containing some macros from `doc/tame.texi' and
some from Liza's `doc/macros.texi'.
* doc/tame.texi: Adjust position of header comment. Include `config.texi'
and `macros.texi'. Add devnotice to header. Strip out macros.
(menu): Add `Concept Index' and conditional `Developer Notes Index'.
(Concept Index, Developer Notes Index): New nodes (latter conditional).
I want this manual to be useful both to developers and users of TAME,
so this distinction needs to be made clear.
* doc/tame.texi (Preprocessor): chapter=>appendix.
* src/graph.texi: Top to appendix and raise subsections.
* src/symtable.texi: Top to appendix.
This is an assumption that's existed since the Summary Page was first
devised---that all vectors have at least one value. This is because the
bucket (originating from Liza) always has at least one value in its vectors.
Of course, we still have a problem in that the Summary Page initializes
everything to have a single value by default, and that's still the
case. But this will at least allow for things _outside_ the Summary Page to
provide an empty array. I'll have to address the Summary Page separately,
and that's going to be difficult, since we don't really want to change the
behavior across the board.
* src/current/compiler/js.xsl (set_defaults): Default max index to 0 if
`length' is unavailable, rather than 1.
The previous length check existed as a really bad array check (before
Array.isArray was a thing). This has been broken since Nov 2012.
The problem manifests itself when you want an empty array. We then have:
[] => [[]] => [DEFAULT_VALUE]
* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Use
`Array.isArray' in place of length check.
TODOs shouldn't be stored here, and they will get out of sync.
* Makefile.am (tame_TEXINFOS): Remove todo.texi.
* tame.texi: Remove include and menu entry.
* todo.texi: Remove file.
This is a BC break since this generates assertions by default. To maintain
BC, set `@allow-zero@' and `@allow-negative@' to `true' in existing template
applications.
* core/insurance.xml
(assert_ignore_premium_zero, assert_ignore_premium_negative): New params.
(_premium_): Generate assertions.
[@allow-zero@, @allow-negative@]: New params.
lsimports will be able to be used to replace the last remaining Ant script
that generates depfiles.
* build-aux/check-coupling:
* build-aux/lsimports: New files.
This allows customizing from the command-line what suppliers should be
checked. This motivation for this is both to run as part of a distributed
pipeline (where each supplier may be built individually), and for during
development of a single supplier.
BC BREAK: Note that this will now check for `package' in the test path for
UI tests. To keep the old directory around, a symlink of `packages' to `ui'
would suffice.
* build-aux/Makefile (SUPPLIERS, suppliers_strip): New variables.
(check-am): BC-BREAK: Build and check only requested suppliers.
* build-aux/progtest-runner: BC-BREAK: First argument is now test directory
and all remaining arguments specify the supplier XML files to check.
We want to be able to build the UI independently of the
suppliers. Historically, this did not provide much of a benefit, but this
change allows us to build independently as a job in a distributed pipeline,
and allows testing out the UI when rating is unneeded.
* build-aux/Makefile.am (program-ui): Remove `standalones'.
This target has not been used for years.
* build-aux/Makefile.am (program-ui-immediate): Remove target.
(program-ui): Use dependency of old `program-ui-immediate'.
(.PHONY): Remove `program-ui-immediate'.
The difference is described here:
http://www.saxonica.com/html/documentation/using-xsl/embedding/
And s9api here:
http://www.saxonica.com/html/documentation/using-xsl/embedding/s9api-transformation.html
* Makefile.am (DSLC_CLASSPATH): Export for submakes.
* configure.ac (DSLC_CLASSPATH): Prefix with SAXON_CP.
* rater/rater.xsd (classNameType): Increase length 50=>75 (generated
identifiers can now exceed that, it seems).
* src/current/rater.xsd: Likewise. These files need to be combined.
* src/current/src/Makefile (CLASSPATH): Set to DSLC_CLASSPATH.
* src/current/src/com/lovullo/dslc/DslCompiler.java: Update imports.
(DslCompiler)[_DslCompiler]: New members _processor and
_xsltCompiler. Convert to s9api.
Note that such files may not actually exist, which is why `nullglob' is set
and the `for' loop is used.
* build-aux/Makefile.am (SHELL): Set `nullglob'.
(program-data-copy, lvroot): Copy srv/!(rater).js to destination JS paths.
This has been broken for years. I don't object to fixing it, it's just that
I have better things to do right now and we've gotten complaints about it;
no use in keeping around something that's broken if there's no desire to fix
it. Workaround: refresh the page.
This does keep around the reset logic because it is actually used in other
places.
* src/current/include/entry-form.xsl (entry-form)[lv:package]: Remove reset
button.
* src/current/include/entry-form.js (clearTestCases): Remove broken function
call `Prior.setPriorMessage(null)'.
Mike Gerwitz