Commit Graph

477 Commits (4f3dfc3bc7f0a9f3a5bbf2c0483bc0b0fee9fc2e)

Author SHA1 Message Date
Mike Gerwitz 4f3dfc3bc7 build-aux/Makefile.am: (program.expanded.xml): .version.xml dependency 2019-05-07 12:03:30 -04:00
Mike Gerwitz c54a87e3e3 progtest (HtmlConsoleOutput): Correct indexing
Caused by previous commit.
2019-05-06 16:59:41 -04:00
Mike Gerwitz aaf7b47e9e progtest (AsyncTestRunner): Fix reporter line breaking
It was adding a count and a line break after the first test run.
2019-05-06 15:53:20 -04:00
Mike Gerwitz 5706ab4bef summary: Fill timestamp_* param values automatically
These exist because TAME is nondeterministic, so all state must be passed
into it.  But it's inconvenient to have users have to manually fill in
dates, so we derive them from the environment unless they are set.

* src/current/scripts/entry-form.js (fillTimeValues): New function.
  (rater): Use it.
2019-04-19 11:46:29 -04:00
Mike Gerwitz f270220b11 build-aux/csvm2csv: Propagate csvm-expand exit status
csvm2csv was not failing when csvm-expand exited with a non-zero
status.  Further, the tests were written incorrectly to account for this.

* build-aux/csvm2csv: Set `pipefail' option.
* build-aux/test/test-csvm2csv: Fix tests.
2019-04-09 10:57:59 -04:00
Mike Gerwitz 4c61dfb1cc csvm: Permit all whitespace (including tabs)
While tabs aren't desirable, users that are not developers will be modifying
these files, and so we need to be permissive in what we want to
accept.  That doesn't mean that we need to forego occasional formatting, though.
2019-04-08 15:17:00 -04:00
Mike Gerwitz 6f5796238a tame: Create guard parent directory
It may not exist on certain systems (e.g. build containers).
2019-04-04 14:52:36 -04:00
Mike Gerwitz 1a35232bd8 Parallel build support
tamed was originally designed with support for parallel builds in mind, but
I hadn't completed that work because we didn't have enough hardware that
we'd benefit strongly from it.  That has since changed.

tamed will now spawn additional runners as needed to fulfill requests, which
works around the issue of not knowing how many jobs GNU Make is going to try
to do at once.

There were a couple minor dependency fixes/workarounds for now in the
Makefile, but otherwise everything appears to be working great.
2019-04-04 14:41:07 -04:00
Mike Gerwitz 7b7cf13607 build-aux/csvm-expand: {orig=>src} local arg typo fix
This does not affect its functionality.
2019-04-02 11:05:03 -04:00
Mike Gerwitz 9a1f916486 build-aux/csvm-expand: Spawn only one date and memoize
A table with a couple hundred thousand rows was taking minutes to
generate.  This gets it down to a few seconds.

* build-aux/csvm-expand (parse_date): New function.
  (parseline): use it.
2019-04-02 10:58:12 -04:00
Mike Gerwitz 3d07597f7c core/insurance (_credit_, _debit_)[@allow-zero@]: Add 2019-03-19 13:19:33 -04:00
Mike Gerwitz 6f9a95d306 progtest: Fail on unknown param as input 2019-02-27 11:44:35 -05:00
Mike Gerwitz c062cc5a5c progtest: Check inputs against known params
This aims to prevent needlessly wasted time debugging a non-working test
case, and to avoid writing incorrect test cases that happen to succeed even
though their inputs aren't properly defined.

For example, a common error is to use the name of a bucket field rather than
the name of the param that it maps to.

* progtest/src/TestRunner.js (_verifyKnownParams): New method.
  (_tryRun): Use it.
* progtest/test/TestRunnerTest.js: New test case.  Modify existing test
   cases to define used params.
* progtest/test/_stub/program.js (exports.rater.params): Declare used param.
2019-02-26 11:10:25 -05:00
Mike Gerwitz 602a77443f compiler: Expose params via compiled rater function
* src/current/compiler/js.xsl (compiler:exit-rater)[lv:package]: Expose
    `params' publicly on the rater function.
2019-02-26 11:09:23 -05:00
Mike Gerwitz c5a99e594d Another round of xmlo compilation performance enhancements
This reduces overall build times for one of our systems by ~50% by
addressing a lot of the low-hanging fruit for compilation of object
files.  There is much more work to be done, and the addition of maps added a
little bit of a mess that will be abstracted in future commits once I'm done
surveying the possible improvements that can be done.
2019-02-20 09:42:19 -05:00
Mike Gerwitz 5714bfb96b symtable: Substantial performance improvement in processing
This further improves performance of the symbol table processing.  The next
step will be to address how symbols are handled on a more intimate level,
since it's a huge mess atm.  But I'll save that for later, after the
low-hanging fruit has been resolved.

* src/current/include/preproc/symtable.xsl (preproc:sym-discover): Use
    `for-each-group' in place of `preceding-sibling'.  Aggressive use of
    maps for geneating the `dedup' sequence, which is a mess.
  (preproc:symtable-process-symbols): Additional maps to avoid
    preceding-sibling and following-sibling selectors (O(n²)=>O(n)).
2019-02-20 02:03:20 -05:00
Mike Gerwitz 16749a9a45 fragment: Iterate over document and use symtable map
Same concept as previous commits: rather than iterating over the symbol
table and scanning the tree for the matching node, iterate over the document
and look up from a symbol map: O(n²) => O(n).

This gives a respectable performance boost to compilation of certain
packages (best improving packages with many classifications or rate blocks).

* src/current/compiler/fragments.xsl (@xmlns:xs, @xmlns:map): New namespace
    declarations.
  (preproc:compile-fragments): Generate `preproc:fragment' nodes and match
    on document rather than symbols.
    [lv:package]: Generate map and tunnel it.
* src/current/compiler/js.xsl (compile)[lv:classify, lv:match]: Use
    symtable-map.
  (compile-class-condition)[lv:rate]: Likewise.
  (compile-cmatch)[lv:rate]: Likewise.
2019-02-20 00:26:32 -05:00
Mike Gerwitz dae1990a00 symtable: Speed up processing a bit
This uses the same map strategy (and same duplicate code) as previous
commits, but this one generates a map for two separate tables.

There is more room for improvement, but this cuts down on the time a
lot.  Also keep in mind that this is performed multiple times (once per
pass), so it's still worth revisiting.  Performance is still very poor for
very large (many thousands of symbols) symbol tables.

The next slowest part appears to be the fragment compilation.  I'm nearing
the end of the low-low-hanging fruit for maps.  The /common/gl package
mentioned in previous commits that previously took over a minute to compile
now compiles in 20s as of this commit on equivalent hardware.

* src/current/include/preproc/symtable.xsl (@xmlns:map): New namespace
    declaration.
  (preproc:symtable-process-symbols): Create map for `cursym' and
    `extresults'.  Use it.  Remove unused `dup'.  Output message when
    done (another is output slightly later on in the process).
2019-02-20 00:10:42 -05:00
Mike Gerwitz 063e68b3d0 validate: Use map for symbol table
This is the first step to improving the map.  Note that this duplicates the
symbol table generation code that's used in a few other places
now---that'll be cleaned up in future commits once I have a better idea of
all the places this will be used and try to move it to a higher level.

* src/current/compiler/validate.xsl (@xmlns:xs, @xmlns:map): New namespace
    definitions.
  (lvv:validate)[lv:package]: Generate symbol table map.  Tunnel to
    templates.
    [c:apply[@name], lv:classify[@as]//lv:match, lv:match[@value]]
    [c:*[@name or @of], c:apply/c:arg[@name], lv:rate/lv:class]: Use it.
2019-02-18 15:42:22 -05:00
Mike Gerwitz c077a71402 resolv-syms: Simplify dimension resolution
The existing code was not only complex (because of XSLT 1), but mostly
unnecessary.  We don't need to consult remote symbol tables at all anymore.

This shaves off an additional few seconds on large packages.

* src/current/include/preproc/package.xsl (preproc:resolv-syms)[preproc:sym]:
  Only consult local symbol table.  Simplify max dimension calculation.
2019-02-18 11:56:17 -05:00
Mike Gerwitz d2ab6e1149 resolv-syms: Generate maps for symtable and dependency lists
This is a first step (low-hanging-fruit kinda thing) for improving the
performance of symbol resolution, where the compiler has to figure out the
dimensions of a symbol by first resolving its dependencies,
recursively.  This is approximately an O(n³) polynomial-time algorithm _per
recursive step_.  Yikes.

This is traditionally where dynamic programming methods would be used, but
that's considerably more difficult in a immutable languages like XSLT, so
I'll do my best without.  (Saxon does offer some support for mutability, but
I'd prefer to avoid it if possible.)

This first change improves performance 30--40%.  For example, on two large
packages we have, build times drop from 55s to 35s and from 1m42s to 1m13s
respectively.

Good start, but much more to be done!

* src/current/include/preproc/package.xsl (preproc:resolv-syms)[lv:package]:
    Compute maps for preproc:symtable and preproc:sym-deps at each recursive
    step.  Pass along via tunneling.
  (preproc:resolv-syms)[preproc:sym]: Use them.

DEV-4354
2019-02-18 11:56:17 -05:00
Mike Gerwitz 09eb442c63 linker: Index root package symbol table
This only saves 1--2s on a 30s run, but I want to move into this direction,
so it'll simplify future refactoring if I just add it.  Small changes like
these will accumulate, too.

* src/current/compiler/linker.xsl (l:orig-package, l:root-symtable-map): New
    variables.
  (l:resov-extern): Use it.
2019-02-18 11:56:17 -05:00
Mike Gerwitz 14dc534709 test/symtable/symbols.xspec: Fix failing test
A bunch of failing pipelines apparently wasn't obvious to me.  And shame on
me for not running these locally; I forgot that the part of the system that
I touched had tests.

This was broken by b6cfdb4221.
2019-02-18 11:29:02 -05:00
Mike Gerwitz 26249f8dbb core: Add _vfilter-mask_
* core/test/core/suite.xml: Import `vector/filter'.
* core/test/core/vector/filter.xml: New package.
* core/vector/filter.xml (_vfilter-mask_, _vfilter_mask): New template, function.
2019-02-14 15:05:49 -05:00
Mike Gerwitz 279245d168 core: Add missing _minreduce_ @isvector@ test
core/test/core/vector/minmax.xml: Add missing @isvector@ test.
2019-02-13 16:10:26 -05:00
Mike Gerwitz 46b7c234dd core: vector/minmax/_minreduce_: New template
* test/core/suite.xml: Import `test/core/vector/minmax'.
* test/core/vector/minmax.xml: New package.
* vector/minmax.xml (_minreduce_, _minreduce): New template, function.
2019-02-13 14:38:08 -05:00
Mike Gerwitz bd73ea0121 Strip xsl: namespace prefix from most files
Still left are files that I don't want to deal with testing right now.
2019-02-07 14:36:18 -05:00
Mike Gerwitz e022a3133d Copyright year simplification and update to Ryan Specialty Group
This now uses year ranges, which I'll update annually.

This also renames "R-T Specialty" to "Ryan Specialty Group".  The latter is
the parent company of the former.  I was originally employed under the
former when LoVullo Associates was purchased, by I now work for the parent
company.
2019-02-07 13:23:09 -05:00
Mike Gerwitz 7862eef62e summary: Accommodate now-missing dependency lists
The previous commit made dependency lists optional for certain symbols.  The
Summary Page needs to be updated to permit such a thing.

The whole Summary Page needs aggressive refactoring, though, so this doesn't
bother checking for `no-deps' to see if this is a bad thing.

* src/current/summary.xsl (typeset-final)[preproc:sym-ref]: Permit missing
    symbol dependencies.
  (lv:param|lv:const|lv:item): Likewise.
2019-02-07 13:11:31 -05:00
Mike Gerwitz b6cfdb4221 depgen: Quadratic=>linear-time algorithm
This is a significant performance improvement for dependency
generation (which is responsible for building the dependency graph for a
package).

The previous algorithm ran in O(n²) time: it would iterate over the given
symbol table, and for _each_ symbol, do a linear scan of the entire document
to search for the corresponding source block.  This resulted in explosive
depgen time for larger packages.

This makes the algorithm run in O(n) by:
  - Using an XSLT 3 map for the symbol table for O(1) lookups; and
  - Iterating over the _document_ a single time rather than the symbol
    table, referencing the symbol table as needed (in O(1) time).

There are other parts of the system that can benefit from these same
improvements.  This is important, since we need to be able to handle many
thousands of symbols efficiently.

* src/current/compiler/linker.xsl (l:depgen-sym): Recognize smybol `no-deps'
    property, permitting missing dependencies.  This allows us to avoid
    creating nonsense nodes just to satisfy the linker, while still allowing
    the linker to perform essential checks to defend against compiler bugs.
* src/current/compiler/map.xsl (lvmc:stub-symtable): Set @no-deps on
    `___head' and `___tail' symbols.
  (lvmc:mapsym): Set `no-deps' as appropriate on map symbols.
  (preproc:depgen)[lvm:map[@from]]: Generate `preproc:sym-dep' node, which
    is now expected by the depgen process.
  (preproc:depgen)[lvm:map[*]]: Likewise.
  (preproc:depgen)[*[@lvmc:type='retmap']//lvmm:map[@from]]: Remove
    unnecessary template.
  (preproc:symtable)[lvm:map[@value]]: Pass `no-deps' to `lvmc:mapsym'.
* src/current/include/depgen.xsl (preproc:depgen)[preproc:symtable]: Create
    and use XSLT 3 map in place of `preproc:symtable' tree.  This allows for
    constant-time lookups.  Provide to templates via tunnelling.  Use it in
    place of exiting tree references.  Process source tree rather than
    iterating over symbol table.
  (preproc:depgen)[lv:rate, c:sum[@generates], c:product[@generates],
    lv:classify, lv:function/lv:param, lv:function, lv:typedef]: Produce
      `preproc:sym-dep' nodes (which was previously done while iterating
      over the symbol table).
  (preproc:depgen)[preproc:sym]: Remove all such processing, since we no
    longer iterate over the symbol table.
  (preproc:depgen)[c:value-of]: Use symtable map.
  (preproc:depgen-match): Likewise.
  (preproc:depgen)[lv:union]: Modify to handle changes to lv:typedef
    template.
  (preproc:depgen)[text()]: Remove and replace with `node()'.
* src/current/include/preproc/package.xsl (preproc:resolv-syms): Remove
    logging of symbol resolution.  This has a slight performace impact since
    there is a lot of output.
* src/current/include/preproc/symtable.xsl
  (lv:function/lv:param, c:let/c:Values/c:value): Set `no-deps'.
* src/symtable/symbols.xsl: Add documentation of `no-deps'.
  (preproc:symtable)[lv:meta]: Set `no-deps'.
2019-02-07 11:39:50 -05:00
Mike Gerwitz 8e7a946127 vector/table: Add comparison operators 2019-02-04 12:21:05 -05:00
Mike Gerwitz 11109d4361 core: Add _where-*_ query predicate templates
These provide a more pleasent abstraction than having to reference CMP_OP_*
constants.

* core/test/core/vector/interpolate.xml: {t:when=>t:where-eq}.
* core/test/core/vector/table.xml: Likewise, but using the other variants
    where appropriate given the value of `@op'.
* core/vector/interpolate.xml: Likewise.
* core/vector/table.xml (_when_, _where_): Rename former to latter and
    provide deprecation warning.
  (_when-lt_, _when-lte_, _when-gt_, _when-gte_): Add abstractions.
* src/current/rater.xsd: Permit template variable as tenplate name.
2019-02-04 10:22:46 -05:00
Mike Gerwitz 36a3e348b6 core: Add comparison operators for table query predicates
This is fairly primitive support and it completely sidesteps the bisect
algorithm for now.  The next commit will abstract this a little bit further
to make it less awkward to use.

* core/test/core/vector/table.xml: New test cases.
* core/vector/filter.xml (CmpOp): New typedef.
  (mfilter): Document that bisecting will not happen unless `CMP_OP_EQ'
    is used.  Implement that restriction.
    [op]: New parameter.  Provide it to `mrange'.
  (_mfilter, _mrange_cmp): Rename from `_mfilter'.  Implement new comparison
    check based on `op'
    [op]: New argument.
* core/vector/table.xml (_when_)[@op@]: New param.  Add it to the produced
    vector.
  (_mquery): Unpack op (from `_when_') in call to `mfilter'.
2019-02-04 10:22:46 -05:00
Mike Gerwitz 74f8b56fcc Use some modern shorthands for core/vector/{table,filter}
Just trying to clean up a little as I go to start to make it easier
to understand.

* core/vector/filter.xml: Use _when-*_ templates and c:recurse.
* core/vector/table.xml: Likewise.
2019-02-04 10:22:46 -05:00
Mike Gerwitz 9af38261b9 calc.xsd: Permit template applications within c:value-of
* src/current/calc.xsd (valueType): Permit ##other nodes.
2019-02-04 10:22:46 -05:00
Mike Gerwitz c68c2f41d5 core/vector/table: Add specification for main templates
* core/test/core/suite.xml: Import core/test/core/vector/table.
* core/test/core/vector/table.xml: New specification.
2019-02-04 10:22:46 -05:00
Mike Gerwitz a35844f3fb depgen: Do not recurse into templates
Same logic, more efficient implementation.

* src/current/include/depgen.xsl (preproc:depgen): Stop at lv:template.
2019-02-02 23:19:40 -05:00
Mike Gerwitz 07e5dbd94b Add expand-barrier and skip-child-expansion
It's going to be like TeX before you know it... ._.

* src/current/include/preproc/package.xsl (preproc:tpl-check)
  [lv:template|lv:const|lv:typedef|lv:param-copy]: Add lv:param-copy.
* src/current/include/preproc/template.xsl (preproc:apply-template)
 [lv:expand-barrier, lv:skip-child-expansion]: New expansion control
   structures.
2019-02-01 16:01:56 -05:00
Mike Gerwitz 7ac4c1ce9d Template variable expansion on (lv:param-value|lv:param-inherit)/@name
This allows for dynamically generated metadata names.

* src/current/include/preproc/template.xsl (preproc:apply-template)
  [lv:param-meta]: Expand @name.
  [lv:param-inherit]: Expand @meta.
2019-02-01 16:01:56 -05:00
Mike Gerwitz f719d391c7 Default short-hand constant description to c:value-of/@label
This is a much more useful description if present.

* src/current/include/preproc/macros.xsl (preproc:macros)[c:value-of...]:
    Default generated constant description to @label.
2019-02-01 16:01:56 -05:00
Mike Gerwitz 73d691273e core: Replace all occurrences of c:{set=>vector}
The former is deprecated and never made any sense at all.
2019-02-01 16:01:56 -05:00
Mike Gerwitz b725963722 Add c:vector as c:set alternative
The term "set" is all wrong---it is actally intended to be a vector, and can
absolutely have duplicate elements (and often does).

* src/current/calc.xsd (vector): Add, recommending in place of `set'.
* src/current/compiler/js-calc.xsl (compile-calc)[c:set|c:vector]:
    Add `c:vector' and provide deprecation notice for `c:set'.
* src/current/include/calc-display.xsl (c:set|c:vector): Likewise.
2019-02-01 16:01:56 -05:00
Mike Gerwitz e4ccf3e90a test/spec: Work around expand-sequence bug
* core/test/spec.xml (_describe_): Enclose aggregate classification in a
  series of nested expand-sequence to work around bug (described in
  comment), which was causing test cases to not be compiled.
2019-02-01 16:01:56 -05:00
Mike Gerwitz 304faa1f07 summary: Remove rate-group processing
* src/current/summary.css (.rate-group, .rate-groups): Remove.
* src/current/summary.xsl (gen-menu): Remove rate-group processing.
  (rate-group-title): Remove.
  (lv:rate-group): Remove.
2019-02-01 16:01:56 -05:00
Mike Gerwitz 22aa59b5cf map: Properly default value in translation
A better option is to pre-process all inputs, but I need a quick
fix to my stupidity.  0||""==="".

* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Correct oval default.
2019-02-01 16:01:56 -05:00
Mike Gerwitz c7fec6a240 csv2xml: Import /rater/core{=>/base} directly
/rater/core is being removed.
2019-02-01 00:36:48 -05:00
Mike Gerwitz 9070d97e87 doc (Core Concepts): Initial stub section
I wanted to get this section started so that I can easily add to it when I
have small bits of time to do so.  Our documentation needs to improve.

* doc/Makefile.am (tame_TEXINFOS): Add `concept.texi'.
* doc/concept.texi: New file.
* doc/preproc.texi: Remove accidentally added input line.
* doc/tame.texi (menu): Add `Core Concepts' node.
2019-01-30 13:45:43 -05:00
Mike Gerwitz 01a420fd81 Revert "set_default: Allow empty vectors"
I need to revert this for now because it breaks YAML test cases.  The proper
fix is a more expressive type system with dependent types that would allow
it to know the proper number of indexes to initialize relative to other
inputs.  I wanted to implement this anyway to help catch iteration-related
bugs.

I'm tabling this for now, though, since I have other things that I need to
work on.

This reverts commit 4406cbe553.
2019-01-30 13:45:17 -05:00
Mike Gerwitz a985cf1f23 doc ({About=>Using} TAME): {about=>usage.tex}
* doc/Makefile.am (tame_TEXINFOS): {about=>usage}.texi.
* doc/tame.texi: Include {about=>usage}.texi
* doc/about.texi: Rename file.
* doc/usage.texi: New file (renamed from about).
2019-01-30 13:45:15 -05:00
Mike Gerwitz 0c67e85676 doc: Add cindex entries for existing About
* doc/about.texi: Add miscellaneous entries.
2019-01-30 13:44:43 -05:00