Commit Graph

503 Commits (e52dd458722b70033c1068605c6749612b19e0cd)

Author SHA1 Message Date
Mike Gerwitz e52dd45872 tamer/rustfmt (max_width): Set to 80 2019-11-27 09:15:15 -05:00
Mike Gerwitz c4a8eac59e Makefile.am: Clean up currently-unused path_ vars
Cargo handles it for us.
2019-11-20 10:11:00 -05:00
Mike Gerwitz 7412a8934c tameld: Placeholder binary 2019-11-20 10:11:00 -05:00
Mike Gerwitz f72ff973a7 Makefile.am (all): {cargo=>@CARGO@}
Typo.
2019-11-20 10:11:00 -05:00
Mike Gerwitz f0ca5c60c9 Makefile.am (doc, html): New documentation target 2019-11-20 10:11:00 -05:00
Mike Gerwitz 0ab2b09dc6 Scaffolding for TAMER 2019-11-19 15:30:10 -05:00
Mike Gerwitz aff58dab63 .gitlab-ci.yml (build): Use bootstrap script
No use in maintaining this stuff in two places.
2019-11-18 14:15:21 -05:00
Mike Gerwitz d20e2bc78a tamer: Integrate into normal build process
Rust is now expected to be installed in the base image.
2019-11-18 14:15:07 -05:00
Mike Gerwitz a2478938b8 .gitlab-ci.yml (build): Clean up script
This has since been moved into the Docker image.
2019-11-18 14:06:55 -05:00
Mike Gerwitz d0208bf89b .gitlab-ci.yml (image): Make variable (BUILD_IMAGE)
We moved to an internal container registry so that we do not have to rely on
DockerHub.  Since TAME is a public project, this will allow our
configuration internally to vary from a public configuration.
2019-11-18 14:06:55 -05:00
Mike Gerwitz 8e241218b7 tamer build as part of pipeline and bootstrap 2019-11-18 14:06:54 -05:00
Mike Gerwitz fd1a5837ba TAMER: Initial commit 2019-11-18 14:05:47 -05:00
Joseph Frazer d9ecbd4e2c [DEV-6370] Package changes
Merge branch 'jira-6370'

* jira-6370:
  [DEV-6370] Allow recursive conditionals
  [DEV-6370] Pass in the $line_code rather than using it from the contract
2019-11-04 11:35:23 -05:00
Joseph Frazer 0fadbe8e8a [DEV-6370] Allow recursive conditionals
If an `lvm:if` is immediately followed by another 'lvm:if`, both should
be used to create the conditional. The existing code wouild only "select
the nearest condition".
2019-11-01 10:23:19 -04:00
Joseph Frazer cbe32aff72 [DEV-6370] Pass in the $line_code rather than using it from the contract
The LOB being passed into the function was being ignored and instead it
was pulling it from the contract object. With Package, this caused all 3
LOB to be "COMMPKGE" rather than the correct LOB being processed at the
time.

Going forward, one cannot `map` or `pass` to "line_code" as it will be
considered a reserved word.

Co-Authored-By: Jim Grundner <james.grundner@rtspecialty.com>
2019-11-01 10:23:19 -04:00
Mike Gerwitz ad7d585d37 TAME: {Adaptive=>Algebraic}
Just adapting the backronym a bit to be more accurate.  I've been meaning to
do this for over a year.
2019-10-28 13:15:37 -04:00
Mike Gerwitz 3ef6571922 Provide friendly lv:param-typedef-lookup failure for duplicate item values
The real solution is to disallow typedefs from getting into this state to
begin with, but I don't have time for that right now.
2019-10-25 13:56:47 -04:00
Mike Gerwitz e97f7a75c9 core/test/vector (_define-vector_): Require description
We want things to require documentation when possible.
2019-10-17 09:20:15 -04:00
Mike Gerwitz 39c7161cca vector/define: New package introducing _define-vector_
This is intended to work around the issue of defining arbitrary vectors
outside of a c:let.
2019-10-17 09:16:45 -04:00
Mike Gerwitz ce0e31032b core: ui: Add _match-ui-*_ templates
These are analaogus to the _match-*_ counterparts, but they convert @on@
into the question param and check against the applicability of the question.
2019-08-13 16:46:06 -04:00
Mike Gerwitz 8005268a1b lv:param-typedef-lookup: New preprocessor directive 2019-08-06 15:31:48 -04:00
Mike Gerwitz a58243c403 core/ui (_match-ui-applicable_): Account for applicability
It doesn't makes sense to consider a question to be set if it's not even
applicable.  This also helps to remove a bunch of duplicate code where these
templates are being used.
2019-07-30 14:35:05 -04:00
Mike Gerwitz 3e13b733c4 core/vector/cmatch.xml (_classify-vector_): New template
This is the analog of _classify-scalar_.
2019-07-29 14:51:38 -04:00
Mike Gerwitz 90bedc20f8 map: Nested value support for input map
For example: meta:foo.bar.baz.

DEV-3871
2019-06-14 11:02:18 -04:00
Mike Gerwitz 13ed4cd7dc Clean up extclass remenants
This is left over from f2db9f1268, in which I
should have cleaned all of this up.  One of our developers was hitting the
removed warning, which isn't necessary since the concept of a separate
"classifier" is no longer a thing after the aforementioned commit.

* rater/rater.xsd (no-extclass, no-extclass-keeps): Remove.
* src/current/rater.xsd: Likewise.  (I really need to deduplicate these.)
* src/current/compiler/js.xsl (compiler:entry-rater): Remove inaccurate
    comment (genclasses is used for other things).
* src/current/include/depgen.xsl (preproc:depgen-match): Remove error
    checking for pulling in non-external classes (this is the error that the
    developer hit that is no longer needed).
* src/current/include/preproc/eligclass.xsl (preproc:sym): Remove
    `@extclass' predicate.  Remove portion of comment.
* src/current/include/preproc/expand.xsl: Remove ancient footnote that
    even references an old internal rater!
* src/current/include/preproc/macros.xsl (preproc:class-groupgen): Remove
    external propagation.
* src/current/include/preproc/symtable.xsl (preproc:symimport): Remove
    extclass checks and propagation.
  (preproc:symtable)[lv:rate]: Remove external propagation.
    [lv:classify]: Likewise.
* src/current/include/preproc/template.xsl (preproc:inline-apply): Remove
    external sym metadata support.
2019-05-22 12:57:35 -04:00
Mike Gerwitz c888e17e97 Revert "build-aux/Makefile.am: (program.expanded.xml): .version.xml dependency"
Now I remember why I didn't do this: it forces a rebuild of
program.expanded.xml every build.

This reverts commit 4f3dfc3bc7.
2019-05-07 14:04:11 -04:00
Mike Gerwitz 4f3dfc3bc7 build-aux/Makefile.am: (program.expanded.xml): .version.xml dependency 2019-05-07 12:03:30 -04:00
Mike Gerwitz c54a87e3e3 progtest (HtmlConsoleOutput): Correct indexing
Caused by previous commit.
2019-05-06 16:59:41 -04:00
Mike Gerwitz aaf7b47e9e progtest (AsyncTestRunner): Fix reporter line breaking
It was adding a count and a line break after the first test run.
2019-05-06 15:53:20 -04:00
Mike Gerwitz 5706ab4bef summary: Fill timestamp_* param values automatically
These exist because TAME is nondeterministic, so all state must be passed
into it.  But it's inconvenient to have users have to manually fill in
dates, so we derive them from the environment unless they are set.

* src/current/scripts/entry-form.js (fillTimeValues): New function.
  (rater): Use it.
2019-04-19 11:46:29 -04:00
Mike Gerwitz f270220b11 build-aux/csvm2csv: Propagate csvm-expand exit status
csvm2csv was not failing when csvm-expand exited with a non-zero
status.  Further, the tests were written incorrectly to account for this.

* build-aux/csvm2csv: Set `pipefail' option.
* build-aux/test/test-csvm2csv: Fix tests.
2019-04-09 10:57:59 -04:00
Mike Gerwitz 4c61dfb1cc csvm: Permit all whitespace (including tabs)
While tabs aren't desirable, users that are not developers will be modifying
these files, and so we need to be permissive in what we want to
accept.  That doesn't mean that we need to forego occasional formatting, though.
2019-04-08 15:17:00 -04:00
Mike Gerwitz 6f5796238a tame: Create guard parent directory
It may not exist on certain systems (e.g. build containers).
2019-04-04 14:52:36 -04:00
Mike Gerwitz 1a35232bd8 Parallel build support
tamed was originally designed with support for parallel builds in mind, but
I hadn't completed that work because we didn't have enough hardware that
we'd benefit strongly from it.  That has since changed.

tamed will now spawn additional runners as needed to fulfill requests, which
works around the issue of not knowing how many jobs GNU Make is going to try
to do at once.

There were a couple minor dependency fixes/workarounds for now in the
Makefile, but otherwise everything appears to be working great.
2019-04-04 14:41:07 -04:00
Mike Gerwitz 7b7cf13607 build-aux/csvm-expand: {orig=>src} local arg typo fix
This does not affect its functionality.
2019-04-02 11:05:03 -04:00
Mike Gerwitz 9a1f916486 build-aux/csvm-expand: Spawn only one date and memoize
A table with a couple hundred thousand rows was taking minutes to
generate.  This gets it down to a few seconds.

* build-aux/csvm-expand (parse_date): New function.
  (parseline): use it.
2019-04-02 10:58:12 -04:00
Mike Gerwitz 3d07597f7c core/insurance (_credit_, _debit_)[@allow-zero@]: Add 2019-03-19 13:19:33 -04:00
Mike Gerwitz 6f9a95d306 progtest: Fail on unknown param as input 2019-02-27 11:44:35 -05:00
Mike Gerwitz c062cc5a5c progtest: Check inputs against known params
This aims to prevent needlessly wasted time debugging a non-working test
case, and to avoid writing incorrect test cases that happen to succeed even
though their inputs aren't properly defined.

For example, a common error is to use the name of a bucket field rather than
the name of the param that it maps to.

* progtest/src/TestRunner.js (_verifyKnownParams): New method.
  (_tryRun): Use it.
* progtest/test/TestRunnerTest.js: New test case.  Modify existing test
   cases to define used params.
* progtest/test/_stub/program.js (exports.rater.params): Declare used param.
2019-02-26 11:10:25 -05:00
Mike Gerwitz 602a77443f compiler: Expose params via compiled rater function
* src/current/compiler/js.xsl (compiler:exit-rater)[lv:package]: Expose
    `params' publicly on the rater function.
2019-02-26 11:09:23 -05:00
Mike Gerwitz c5a99e594d Another round of xmlo compilation performance enhancements
This reduces overall build times for one of our systems by ~50% by
addressing a lot of the low-hanging fruit for compilation of object
files.  There is much more work to be done, and the addition of maps added a
little bit of a mess that will be abstracted in future commits once I'm done
surveying the possible improvements that can be done.
2019-02-20 09:42:19 -05:00
Mike Gerwitz 5714bfb96b symtable: Substantial performance improvement in processing
This further improves performance of the symbol table processing.  The next
step will be to address how symbols are handled on a more intimate level,
since it's a huge mess atm.  But I'll save that for later, after the
low-hanging fruit has been resolved.

* src/current/include/preproc/symtable.xsl (preproc:sym-discover): Use
    `for-each-group' in place of `preceding-sibling'.  Aggressive use of
    maps for geneating the `dedup' sequence, which is a mess.
  (preproc:symtable-process-symbols): Additional maps to avoid
    preceding-sibling and following-sibling selectors (O(n²)=>O(n)).
2019-02-20 02:03:20 -05:00
Mike Gerwitz 16749a9a45 fragment: Iterate over document and use symtable map
Same concept as previous commits: rather than iterating over the symbol
table and scanning the tree for the matching node, iterate over the document
and look up from a symbol map: O(n²) => O(n).

This gives a respectable performance boost to compilation of certain
packages (best improving packages with many classifications or rate blocks).

* src/current/compiler/fragments.xsl (@xmlns:xs, @xmlns:map): New namespace
    declarations.
  (preproc:compile-fragments): Generate `preproc:fragment' nodes and match
    on document rather than symbols.
    [lv:package]: Generate map and tunnel it.
* src/current/compiler/js.xsl (compile)[lv:classify, lv:match]: Use
    symtable-map.
  (compile-class-condition)[lv:rate]: Likewise.
  (compile-cmatch)[lv:rate]: Likewise.
2019-02-20 00:26:32 -05:00
Mike Gerwitz dae1990a00 symtable: Speed up processing a bit
This uses the same map strategy (and same duplicate code) as previous
commits, but this one generates a map for two separate tables.

There is more room for improvement, but this cuts down on the time a
lot.  Also keep in mind that this is performed multiple times (once per
pass), so it's still worth revisiting.  Performance is still very poor for
very large (many thousands of symbols) symbol tables.

The next slowest part appears to be the fragment compilation.  I'm nearing
the end of the low-low-hanging fruit for maps.  The /common/gl package
mentioned in previous commits that previously took over a minute to compile
now compiles in 20s as of this commit on equivalent hardware.

* src/current/include/preproc/symtable.xsl (@xmlns:map): New namespace
    declaration.
  (preproc:symtable-process-symbols): Create map for `cursym' and
    `extresults'.  Use it.  Remove unused `dup'.  Output message when
    done (another is output slightly later on in the process).
2019-02-20 00:10:42 -05:00
Mike Gerwitz 063e68b3d0 validate: Use map for symbol table
This is the first step to improving the map.  Note that this duplicates the
symbol table generation code that's used in a few other places
now---that'll be cleaned up in future commits once I have a better idea of
all the places this will be used and try to move it to a higher level.

* src/current/compiler/validate.xsl (@xmlns:xs, @xmlns:map): New namespace
    definitions.
  (lvv:validate)[lv:package]: Generate symbol table map.  Tunnel to
    templates.
    [c:apply[@name], lv:classify[@as]//lv:match, lv:match[@value]]
    [c:*[@name or @of], c:apply/c:arg[@name], lv:rate/lv:class]: Use it.
2019-02-18 15:42:22 -05:00
Mike Gerwitz c077a71402 resolv-syms: Simplify dimension resolution
The existing code was not only complex (because of XSLT 1), but mostly
unnecessary.  We don't need to consult remote symbol tables at all anymore.

This shaves off an additional few seconds on large packages.

* src/current/include/preproc/package.xsl (preproc:resolv-syms)[preproc:sym]:
  Only consult local symbol table.  Simplify max dimension calculation.
2019-02-18 11:56:17 -05:00
Mike Gerwitz d2ab6e1149 resolv-syms: Generate maps for symtable and dependency lists
This is a first step (low-hanging-fruit kinda thing) for improving the
performance of symbol resolution, where the compiler has to figure out the
dimensions of a symbol by first resolving its dependencies,
recursively.  This is approximately an O(n³) polynomial-time algorithm _per
recursive step_.  Yikes.

This is traditionally where dynamic programming methods would be used, but
that's considerably more difficult in a immutable languages like XSLT, so
I'll do my best without.  (Saxon does offer some support for mutability, but
I'd prefer to avoid it if possible.)

This first change improves performance 30--40%.  For example, on two large
packages we have, build times drop from 55s to 35s and from 1m42s to 1m13s
respectively.

Good start, but much more to be done!

* src/current/include/preproc/package.xsl (preproc:resolv-syms)[lv:package]:
    Compute maps for preproc:symtable and preproc:sym-deps at each recursive
    step.  Pass along via tunneling.
  (preproc:resolv-syms)[preproc:sym]: Use them.

DEV-4354
2019-02-18 11:56:17 -05:00
Mike Gerwitz 09eb442c63 linker: Index root package symbol table
This only saves 1--2s on a 30s run, but I want to move into this direction,
so it'll simplify future refactoring if I just add it.  Small changes like
these will accumulate, too.

* src/current/compiler/linker.xsl (l:orig-package, l:root-symtable-map): New
    variables.
  (l:resov-extern): Use it.
2019-02-18 11:56:17 -05:00
Mike Gerwitz 14dc534709 test/symtable/symbols.xspec: Fix failing test
A bunch of failing pipelines apparently wasn't obvious to me.  And shame on
me for not running these locally; I forgot that the part of the system that
I touched had tests.

This was broken by b6cfdb4221.
2019-02-18 11:29:02 -05:00
Mike Gerwitz 26249f8dbb core: Add _vfilter-mask_
* core/test/core/suite.xml: Import `vector/filter'.
* core/test/core/vector/filter.xml: New package.
* core/vector/filter.xml (_vfilter-mask_, _vfilter_mask): New template, function.
2019-02-14 15:05:49 -05:00