It's going to be like TeX before you know it... ._.
* src/current/include/preproc/package.xsl (preproc:tpl-check)
[lv:template|lv:const|lv:typedef|lv:param-copy]: Add lv:param-copy.
* src/current/include/preproc/template.xsl (preproc:apply-template)
[lv:expand-barrier, lv:skip-child-expansion]: New expansion control
structures.
This is a much more useful description if present.
* src/current/include/preproc/macros.xsl (preproc:macros)[c:value-of...]:
Default generated constant description to @label.
The term "set" is all wrong---it is actally intended to be a vector, and can
absolutely have duplicate elements (and often does).
* src/current/calc.xsd (vector): Add, recommending in place of `set'.
* src/current/compiler/js-calc.xsl (compile-calc)[c:set|c:vector]:
Add `c:vector' and provide deprecation notice for `c:set'.
* src/current/include/calc-display.xsl (c:set|c:vector): Likewise.
A better option is to pre-process all inputs, but I need a quick
fix to my stupidity. 0||""==="".
* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Correct oval default.
I need to revert this for now because it breaks YAML test cases. The proper
fix is a more expressive type system with dependent types that would allow
it to know the proper number of indexes to initialize relative to other
inputs. I wanted to implement this anyway to help catch iteration-related
bugs.
I'm tabling this for now, though, since I have other things that I need to
work on.
This reverts commit 4406cbe553.
I want this manual to be useful both to developers and users of TAME,
so this distinction needs to be made clear.
* doc/tame.texi (Preprocessor): chapter=>appendix.
* src/graph.texi: Top to appendix and raise subsections.
* src/symtable.texi: Top to appendix.
This is an assumption that's existed since the Summary Page was first
devised---that all vectors have at least one value. This is because the
bucket (originating from Liza) always has at least one value in its vectors.
Of course, we still have a problem in that the Summary Page initializes
everything to have a single value by default, and that's still the
case. But this will at least allow for things _outside_ the Summary Page to
provide an empty array. I'll have to address the Summary Page separately,
and that's going to be difficult, since we don't really want to change the
behavior across the board.
* src/current/compiler/js.xsl (set_defaults): Default max index to 0 if
`length' is unavailable, rather than 1.
The previous length check existed as a really bad array check (before
Array.isArray was a thing). This has been broken since Nov 2012.
The problem manifests itself when you want an empty array. We then have:
[] => [[]] => [DEFAULT_VALUE]
* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Use
`Array.isArray' in place of length check.
The difference is described here:
http://www.saxonica.com/html/documentation/using-xsl/embedding/
And s9api here:
http://www.saxonica.com/html/documentation/using-xsl/embedding/s9api-transformation.html
* Makefile.am (DSLC_CLASSPATH): Export for submakes.
* configure.ac (DSLC_CLASSPATH): Prefix with SAXON_CP.
* rater/rater.xsd (classNameType): Increase length 50=>75 (generated
identifiers can now exceed that, it seems).
* src/current/rater.xsd: Likewise. These files need to be combined.
* src/current/src/Makefile (CLASSPATH): Set to DSLC_CLASSPATH.
* src/current/src/com/lovullo/dslc/DslCompiler.java: Update imports.
(DslCompiler)[_DslCompiler]: New members _processor and
_xsltCompiler. Convert to s9api.
This has been broken for years. I don't object to fixing it, it's just that
I have better things to do right now and we've gotten complaints about it;
no use in keeping around something that's broken if there's no desire to fix
it. Workaround: refresh the page.
This does keep around the reset logic because it is actually used in other
places.
* src/current/include/entry-form.xsl (entry-form)[lv:package]: Remove reset
button.
* src/current/include/entry-form.js (clearTestCases): Remove broken function
call `Prior.setPriorMessage(null)'.
It wasn't until recently that I realized that the default browser font was
being used, since I have mine customized.
* src/current/summary.css (body)[font-family]: Sans-serif font stack.
* src/current/compiler/map.xsl:
(lvmc:gen-input-default): Add argument.
[dim]: New param, defaulting to `$sym/@dim'.
(lvmc:compile)[lvm:map//lvm:from[*]]: Provide appropriate dimension value
to `set_defaults'. Provide compile-time error if nesting of `from'
nodes exceeds what is appropriate for the symbol dimensions.
This fixes a number of obnoxious miscellaneous issues, summarized below.
* src/current/src/com/lovullo/dslc/DslCompiler.java (DslCompiler)[compile]:
Output termination line (DONE) on missing destination path
error. Always output exception message before termination
line (otherwise it won't output to the user). Output termination line
and remove destination file for XSD failure.
This was a bit of a nasty one. Fortunately, this was only used as a
validation, so the code that the compiler produced was still correct.
The problem was that a version of Saxon sometime between 9.5 and 9.8 added
an optimization to eliminate conditionals with no body. Consequently, the
kluge to force the variable to be evaluated was optimized away,
`lvmc:get-symbol' was never called, and no error was ever produced.
This would be best refactored, but that's not something I have time to take
up at the moment priority-wise. This should be future-proof since this
would never be a noop.
* src/current/compiler/map.xsl (lvmc:compile)[lvm:map//lvm:from[*]]: Force
evaluation of `$sym' by ensuring that the condition will not be a noop.
* src/current/c1map.xsl (lvm:c1map): Copy `@namespace' to generated
`lvmp:root'.
* src/current/c1map/render.xsl (lvmp:render)[lvmp:root]: Output
`@namespace' rather than using hardcoded string and dynamic program.
This is a long-standing bug, apparently. The location of this code makes it
difficult to test directly (that is in dire need of correcting), but
fortunately we have a number of tests in systems that use TAME that
indirectly test this.
The problem manifested when a matrix was already in the store, but then a
scalar or vector predicate was considered. Without making the branch that
was modified here, it modified store such that it would always yield a
vector.
* src/current/compiler/js.xsl (anyValue): Consider store dimension when
recursing.
This frees us from requiring a rater/ directory in the working
directory. However, it is important that we continue using it if it
exists, since there are additional things that haven't yet been moved
into the tame repo.
* bin/dslc.in: Provide path to rater/ directory.
* src/current/src/com/lovullo/dslc/DslCompiler.java: Use provided rater/ path.
* build-aux/gen-make: Do not add ".xmlo" suffix for deps with a
trailing `$'.
* src/current/pkg-dep.xsl (lvm:program|lvm:return-map): Append ".xml$" to
dep for map/@src (new dep).
This is a major step toward normalcy---removing the kluge of a build process
that was causing so many issues. Rather than echoing all operations to a
queue file before passing it off to dslc, the new build scripts in `bin/'
are used to invoke tame normally, as needed. This solves all of the current
issues with things not rebuilding when they should. And, as a bonus, tab
completion on targets works.
Sorry this took so long. There wasn't much motivation until we hired so
many people that are suffering from this.
This does a few major things, along with some miscellaneous others:
- Invoke bin/tame directly;
- Merge Makefile.2.in into Makefile.am; and
- Fix up some targets.
* build-aux/Makefile.2.in: Delete file. Mostly merged with Makefile.am.
* build-aux/Makefile.am: Add a bunch of new targets and definitions from
Makefile.2.in. Modify all that previously used .cqueue to now invoke
`$(TAME)' directly. Remove miscellaneous targets for trying to proxy
targets to Makefile.2.
(saneout, _go): Remove definitions.
(.NOTPARALLEL): Add to prevent parallel builds.
(ui/program.expanded.xml)[.version.xml]: Remove dependency for now.
(clean): Also clean generated PHP files. Follow symlinks to clean core.
This is still incomplete (does not clean all rate table stuff).
(suppliers.mk)[xmlo_cmd]: Remove. See `gen-make' and `gen-c1make'.
(lvroot)[summary-html]: New dependency.
(kill-tamed, tamed-die): New targets (former alias of latter) to kill
tamed.
* build-aux/gen-c1make: Generate `$(TAME)' invocation.
* build-aux/gen-make: Likewise. Remove `xmlo_cmd' output. Ignore recursive
`tame' symlink (this can be removed once we clean `rater/' up.
* build-aux/m4/calcdsl.m4 (TAME): Update description to reflect that it
should now be the path to `bin/tame'. Adjust `AC_CHECK_FILE' lines
accordingly.
(tame_needed_ver): Remove. We have been in the same repo as TAME itself
for quite some time. Remove associated code.
(AC_CONFIG_FILES): Remove `Makefile.2'.
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
Perform validation prefore `compile' command rather than a separate
`validate' step. Remove `rm'.
[compileSrc]: Stop echoing command. This was only necessary because of
the previous Makefile klugery; now Make echoes on its own correctly.
These scripts allow the TAME compiler stack to be invoked naturally, rather
than requiring the use of a Makefile today. This will not only allow users
to more easily invoke the compiler, but will also allow us to invoke TAME
naturally from Makefile and remove the klugery that has existed for so
long.
This users a server/client architecture in order to mitigate the startup
cost of the JVM. More documentation will follow.
Note that there are a bunch of symlinks in rater/---this is a transition
step to allow the build to continue working as it did before, which relies
on a directory structure that exists outside of this repository. This will
be cleaned up in the future.
* .gitignore (bin/dslc): Add ignore for generated file.
* bin/dslc.in: New script to encapsulate Java invocation.
* bin/tame: New script (client).
* bin/tamed: New script (server).
* configure.ac (JAVA_OPTS, DSLC_CLASSPATH, AUTOGENERATED): New variables for
dslc.in. Output bin/dslc.
* rater/README.md: Note that this symlink mess is temporary.
* rater/c1map: New symlink for dslc assumptions.
* rater/c1map.xsl: Likewise.
* rater/calc.xsd: Likewise.
* rater/compile.xsl: Likewise.
* rater/compiler: Likewise.
* rater/dot.xsl: Likewise.
* rater/include: Likewise.
* rater/link.xsl: Likewise.
* rater/standalone.xsl: Likewise.
* rater/summary.xsl: Likewise.
* rater/tame: Likewise (warning: circular symlink).
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
Output `DONE' lines.
This will now automatically build on recursive target `all'.
* Makefile.am (SUBDIRS): Add `src/current/src'.
* src/current/src/Makefile: (.PHONY): Add `all'.
(all): New target. Alias to `dslc'.
Most LaTeX distributions prohibit loading files from parent
directories. This gets around that for PDF builds.
* doc/src: Add symlink.
* doc/tame.texi, src/symtrable.xsl: Use it.
This was throwing a warning in non-ancient versions of Saxon. It does not
need to be there, nor should it be, nor do I know why it was put there.
* src/current/include/preproc/template.xsl (eseq:is-expandable): Remove
@override.
This has a significant performance impact: processing time is cut in about
half and memory usage is reduced by more than 50%. For example, a
package that previously took 30s and 2.1GiB of memory to link now takes
14s and less than 900MiB of memory.
I had tried to perform this optimization a couple years ago but was
thwarted (I think) by the classifier markers. The previous commit did away
with those. I'm encouraged by the gains from the low-hanging fruit.
* src/current/compiler/linker.xsl
(l:process-empty, l:stack-empty): Convert from l:pstack and
l:sym-stack (respectively) to empty preproc:sym sequences.
(l:depgen-process-sym)[preproc:sym]: Append to sequence rather than
outputting new l:sym-stack tree.
Update all annotations and uses accordingly.
This is something that I thought would be useful back in the day when TAME
was in its infancy, but it is not important. Rather than having the linker
spend time trying to figure out what symbols belong in the classifier---and
rather than keeping that complexity around---this simplifies things by
making the existing `classify' method simply perform _all_ calculations, and
then yield only the classification portion of the result.
This isn't a problem in practice because, if we only desire the use of a
classifier, then we create a "supplier" that only uses classifications and
has no other dependencies. The end result is, as far as we care, the same.
* src/current/compiler/js.xsl (compiler:entry-rater)[lv:package]: Initialize
`classes' rather than invoking classifier
(compiler:entry-classifier)[lv:package]: Invoke all calculations and
return only classes to provide equivalent behavior.
(compiler:exit-classifier): Post-process classifications from calculation
results, iterating through classmap.
(compiler:classifier-yields-map)[lv:package]: Output all classifications
that are not generated. This differs slightly from the original
implementation in that it includes all non-generated classes rather than
just classes that have a non-generated `@yields'; this distinction is
important since `compiler:exit-classifier' is now using it to produce a
classification result set that doesn't contain all the generated
stuff (since it didn't before, and shouldn't now).
* src/current/compiler/linker.xsl: Update copyright year.
(l:resolv-deps)[preproc:sym[@l:mark-inclass]]: Remove template.
(l:resolv-deps)[preproc:sym...@l:mark-inclass...]: Remove template.
(l:depgen-sym): Set type of result to `element(preproc:sym)', since
`l:mark-inclass' is no longer produced.
[inclass, needs-class-mark]: Remove variables and all instances where
they are used.
(l:dep-aug)[inclass]: Remove param. Stop producing `@inclass' attribute.
(l:link-classifier)[lv:package]: Do not process any dependencies. This
can be removed entirely in the future since it now only produces static
code, which we can perhaps combine with a different block.
(l:link-rater)[lv:package]: Remove mention of `inclass' for dependencies;
all dependencies will now be compiled into this block.
This includes a SHA256 implementation which is _not_ intended for secure
cryptographic operations; see src/js/sha256.js header for more information.
* src/current/compiler/js.xsl (compiler:static): Echo src/js/sha256.js.
[map_method_uppercase, map_method_hash]: New functions.
* src/current/link.xsl: Include dslc-base.xsl.
* src/js/sha256.js: New file.
* src/current/compiler/map.xsl
(lvmc:get-method-func, lvmc:value-ref, lvmc:transformation-wrap): New
functions, partyl extracted from below.
(lvmc:compile)[lvm:map//lvm:from]: Use `lvmc:value-ref'.
[lvm:map//lvm:from/lvm:translate]: Add `[@key]' to match.
[lvm:map//lvm:transform]: New match. Ignore node entirely.
(lvmc:concat-compile): Propagate symtable to `lvmc:compile'.
* src/current/include/dslc-base.xsl (__path-root): New param.
* src/current/src/com/lovullo/dslc/DslCompiler.java
(DslCompiler)[compile]: Resolve TAME root path.
[_transform]: Set it.
DEV-3115
This is all really confusing because this doesn't use the same import
specification as packages; maps got stuck in a partial transition. So,
let's provide some helpful errors rather than silently failing.
* src/current/compiler/map.xsl (preproc:symtable)[lvm:import]:
Error if missing `@path'. Provide more information if `@package' was
provided to help clarify.
We need to cut down on symbol imports as much as possible; the whole system
starts dragging if we are importing thousands of symbols into a single
package.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate,c:*]: Mark
as local if `@preproc:generated`.
* src/current/include/preproc/template.xsl (preproc:macros)[lv:inline-template]:
Mark generated templates as such.
* src/symtable/symbols.xsl (preproc:symtable)[lv:template]: Mark as local if
`@preproc:generated'.
* src/symtable.xsl (symtable:find-duplicates):
Update description. Find duplicates in same package.
* test/symtable.xspec: Update test case accordingly.
* src/current/compiler/js.xsl (compiler:js-number): New function to
remove leading zeroes.
(compile)[lv:const]: Use it.
* src/current/compiler/js-calc.xsl (compile-calc)[c:const]: Use it.
* src/current/compiler/js.xsl (compile-class-condtion)[lv:rate]: Do not
consider @no's in predicate generation when `@preproc:gentle-no' is set.
* src/current/include/preproc/macros.xsl (preproc:macros)[lv:rate-each]: Set
`@preproc:gentle-no' on generated `lv:rate', since the generator handles
`@no' itself.
* src/current/include/preproc/template.xsl
(preproc:gen-param-value)[lv:param-sym-value]: Suppress warning for
missing symbol and yield empty string if `@ignore-missing='true'`.
This ensures that they are compiled into the `consts' object.
* src/current/include/depgen.xsl (preproc:depgen)[lv:typedef]: Include
`lv:enum/lv:item/@name' as dependencies.
The problem with this implementation was that, any time a generator had an
associated generated @yields (which is common), it wouldn't be included in
the summary page.
We can address this in the future. It's not necessarily that it was
incorrect; it's just how the system made use of it.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate]:
Do not mark @preproc:yields-generated symbols as @preproc:generated.
Templates can expand into unexpected places, so sometimes warnings are
inappropriately issued.
* src/current/include/depgen.xsl (preproc:depgen)[lv:template]: Ignore.
[lv:template/lv:param]: Remove (now unnecessary with above).
In order for the cmatch algorithm to work properly, predicates must be
re-ordered on @dim descending.
* src/current/compiler/js.xsl (compile)[lv:classify]: Order all different
dimensions, not just scalars.
any/all blocks within classifications are extracted into their own
classifications. When they have sibling nodes, the extracted
classifications did not have their templates expanded. Ouch.
This is a situation that should never happen (I haven't yet determined
whether or not we should support this type of thing in the future), but when
it does, do something intelligent.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[c:*[@generates]]:
Consider only first ancestor lv:rate.
For example, with template-generating templates, if a reference is unknown
and therefore determined to be blank, an attribute might be completely
removed while the template is being generated. (See the /when package in
core for an example.) That is not good.
* src/current/include/preproc/template
(preproc:expand-template): Add `tpl' tunneling param to
preproc:apply-template application.
(preproc:apply-template)[@*]: Retain param reference if applying template
does not define it.
This is important for template-generating templates.
* src/current/include/preproc/template.xsl (lv:if,lv:unlesS):
Retain conditional nodes verbatim when referenced param is unknown.
* src/current/include/preproc/expand.xsl
(lv:classify): Set `@preproc:yields-generated' if `@yields' was not
previously defined.
* src/current/include/preproc/macros.xsl:
(lv:rate-each): Set `@preproc:yields-generated` if `@yields' was not
previously defined.
* src/current/include/preproc/symtable.xsl:
(lv:rate, lv:classify): Set `@preproc:generated` on symbol if
`@preproc:yields-generated'.
Not time to address styling this at the moment. This code was previously
not reached until recent fixes. And I forgot that this was even
permitted (such complex calculations within matches)!
* src/current/summary.xsl (match-desc): Do not terminate on unknown
match-desc elements.
Rather than producing a syntax error, provide a useful warning and simply
yield 0.
* src/current/compiler/js.xsl (compile)[lv:rate]: Warn and yield 0 when no
calculation is provided in the body.
We've never done this before (thus this bug lasting so many years), but only
because it doesn't really make much sense in practice; this was caught when
writing test cases.
* src/current/compiler/js.xsl (compile)[lv:match]: Compile `consts' instead
of `args' when referencing a constant.
* src/current/include/preproc/template.xsl
(preproc:gen-param-value)[lv:param-add]: Resolve value of @value before
processing numeric expression (to support template variables). Error
when any values are not numeric rather than producing NaN.
This is imperfect, but will work based on current assumptions.
* src/current/include/preproc/expand.xsl (preproc:expand)[c:const,lv:const):
If '@' is found, do not perform expansion.
They corrupt the display. This solution isn't ideal, though.
* src/current/include/calc-display.xsl: Use new `preproc:tex-index'.
(preproc:tex-index): Add function.
* src/current/summary.xsl: Use `preproc:tex-index' for @index display.
* src/current/include/display.xsl (preproc:dim-to-str): New function.
(summary:desc): Use it. Display appropriate generator dimensions as a
string rather than hard-coding to `vector'. Remove yield symbol ref.
Vector length is customarily denoted by |V|.
* src/current/include/calc-display.xsl (c:sum, c:length-of): Use new
notation.
* src/current/summary.xsl: Update N.B. with new notation.
There's no point in having a link to itself. Further, the link was wrong.
* src/current/summary.xsl (gen-equation-details): Remove link from generator
name.
When a symbol is imported as local, and subsequently directly imported using
another package with @export, the symbol needs to be overridden as
non-local.
* src/current/include/preproc/symtable.xsl
(preproc:sym-discover): Deduplicate based on local status.
(preproc:symtable-process-symbols): Retain non-local symbols when only
local version of the symbol exists.
This is a linker bug that I don't have time to debug at the
moment. Hopefully the linker will be re-written in Scheme sometime in the
near future anyway.
* src/current/summary.xsl (preproc:sym-lookup): Take first result of symbol
lookup.
If it is copied an yields an empty value, then it is later converted into a
0, which is wholly incorrect in the case of generators.
* src/current/include/preproc/macros.xsl (preproc:macros)[lv:rate-each]:
Conditionally copy `@dim'.
This is just a temporary workaround for a failed lookup; no time to debug at
the moment.
* src/current/summary.xsl (preproc:sym-lookup): Allow empty sequence as
return value.
If the hash is e.g. `:class:*', it tries to parse it, which is
invalid. There's a better solution to this problem, but I don't have time
to test possibly messing this up right now.
* src/current/scripts/entry-form.js (begin): Ignore JSON parsing errors.
This same problem can occur anywhere---everything needs to be converted to
use the new `preproc:sym-lookup', but I don't have the time at the moment.
* src/current/summary.xsl (preproc:sym-lookup): New function.
(program): New toplevel variable.
(process-match)[lv:match]: Use function.
This optimization (which already existed) is supposed to remove
c:sum/c:product nodes if they contain one or fewer calculations, but it
failed to remove them when inline templates were used (because it creates,
well, a template inline).
Instead, we'll assume that any lv:* nodes handled by the preprocessor will
be okay. In the case of inline templates, they receive a preproc attribute
identifying what created it.
I hope this is a valid assumption. This will be thoroughly tested. If only
we had test cases for the preprocessor.
* src/current/include/preproc/expand.xsl: (preproc:expand)[c:sum,c:product]:
Consider optimization for `lv:*[@preproc:*]'.
The idea is to provide some guidance with how YAML test cases are supposed
to appear.
This just adds to the massive cluster that is the `entry-form.js'.
* src/current/include/entry-form.xsl (entry-form)[preproc:sym]: Add
`entry-testcase-dfn' div.
* src/current/scripts/entry-form.js: Invoke `updateParamTestcaseDfn' in
various event listeners.
(populateBucket): Invoke `updateParamTestcaseDfn'.
(updateParamTestcaseDfn, getParamTestcaseDfnElement): New functions.
* src/current/summary.css: Style `.entry-testcase-dfn'.
This existed in the old summary pages. Since we'll be having certain people
reference ids, they need to be easily visible from somewhere.
* src/current/include/entry-form.xsl (preproc:sym)[entry-form]: Add param
name and link.
* src/current/summary.css: Style it.
Long-standing bug. Didn't often see it because FF performance was
relatively poor for the Summary Pages compared to Chromium until recently,
so I didn't often load the page in it (despite it being my primary browser).
* src/current/summary.css (legend): width:auto.
If nested c:let expressions contained values of the same name, they would
all be rendered, rather than just the one intended.
* src/current/include/display.xsl (do-gen-let-list): Pass let symbol to
`_gen-let-list-item'.
(_gen-let-list-item): Render only values part of context let expression.
[letsym]: New parameter.
This was recently introduced and was noticed because c:let/@name was not
being generated under certain circumstances.
* src/current/include/preproc/expand.xsl (preproc:expand): Continue applying
templates after @dim expansion (rather than copying child nodes).
This (hopefully) prevents metadata from leaking outside of template
expansions where they are not wanted.
* src/current/include/preproc/macros.xsl (preproc:macropass): Strip
cruft (new barrier nodes) on final pass.
* src/current/include/preproc/template.xsl (preproc:expand-template):
Enclose template expansion in preproc:tpl-barrier (which is stripped on
the final pass).
(preproc:apply-template)[lv:param-meta]: Document template. "Hoist"
node (set attribute) if not contained within lv:param-copy.
(preproc:gen-param-value)[lv:param-inherit]: Consider hoisted nodes behind
barrier.
This renders template expansions as part of the calculation breakdown, which
goes a long way to understanding what the system is doing.
This further updates the hover over the breakdown to color-code based on
depth, making it easier to observe the nesting of
expressions. Sub-expressions that are toplevel results of template
expansions (nodes marked with templates) have a thicker border.
* src/current/summary.css: Add template id styling, breakdown colors, and
misc. styling.
* src/current/summary.xsl (ultra-breakdown-set): Enclose label in
span. Output template id, if applicable. Mark fieldset if templated.
This allows for debugging expansions, which are otherwise confusing and
difficult. This will be valuable information for the Summary Pages as
well (see future commit).
* src/current/include/preproc/template.xsl (preproc:expand-template): Mark
toplevel nodes after expansion.
(preproc:mark-tpl-expansion): Add templates.
Cruft left around from the symbol table refactoring long ago.
* src/current/compiler/js.xsl
(compile)[preproc:rate]: Remove template.
[preproc:class]: Remove template.
(compile-rates)[lv:package]: Remove template.
This makes me slightly uncomfortable because I haven't researched why
exactly this was not a problem before. We encountered this issue using an
inline template iterating over the symbol table yielding a `c:value-of',
which is admittedly something that we haven't done before.
Hopefully it's an isolated case.
I diffed the offending object file and the entire linked tax calculator and
they were bit-for-bit identical, so we're probably good.
* src/current/include/depgen.xsl (preproc:depgen): Do not perform dependency
generation within templates.
Other code relied on expand-sequence, but this really should be implicit,
since it is deceptively useless otherwise.
* src/current/include/preproc/template.xsl (preproc:macros): Defer
processing of inline-template with sym-set until symbol table is
available.
The primary motivation for this is to allow for template conditionals.
* src/current/calc.xsd (applyArgs): New group.
(apply,recurse): Use it in sequence.
This allows the result of a rate block to be a matrix; there was previously
no way for a named value to be assigned a matrix unless it was a parameter.
This is a bit of a kluge---the compiler won't discover the proper type
information and won't perform the proper safeguards.
* src/current/calc.xsd (sum)[@dim]: Add attribute.
* src/current/compiler/js-calc.xsl: Add xs namespace.
(compile-calc): Do not perform casting when @dim > 1.
* src/current/include/preproc/expand.xsl
(lv:rate-each): Include @dim in c:sum expansion.
* src/current/include/preproc/macros.xsl:
(c:*/@generates): Use @dim to determine symbol dimensions.
* src/current/include/preproc/expand.xsl: Parse @dim aliases (e.g. "vector",
"matrix").
If a param referenced a typedef that was imported from another package, the
summary page displayed an error rather than rendering the field. This is a
problem for testing using summary pages!
Long-standing bug with a fairly easy fix.
* src/current/include/entry-form.xsl:
(entry-form-field lv:param): Consider correct symbol (was using param
symbol, not typedef).
[typesrc]: Modernize syntax.
[typedef]: Allow for non-root-child elements. The old version of TAME
required toplevel typedefs; that's no longer the case.
This can occur during template expansion if the user is not careful in
ensuring that all generated blocks will have unique names. The error was
not useful: it failed type validation, which results in an internal error in
the compiler (from the user's perspective) and outputs no useful information
to aid in debugging.
* src/current/include/depgen.xsl
(preproc:depgen preproc:sym): User-friendly error if more than one rate
element.
[rate]: Allow multiple rate elements.
This is important to include all terminating classifications, which
include assertions. This is essential now that @keep support has been
removed; this essentially does the same thing, but in a more
sane/strict manner.
* src/current/compiler/linker.xsl (l:depgen)[preproc:symtable]:
Include package-level eligibility class in initial dependency list.
This will hopefully provide a performance boost, and is a cheap
optimization to make. The information it collected was pretty useless
in practice.
* src/current/compiler/js-calc.xsl (compile) [c:*]:
Do not encase calculations with function ancestors in debug
collection functions.
These used to be automatically added via @keep.
* src/current/compiler/linker.xsl (l:depgen): Include meta symbols.
* src/symtable/symbols.xsl (lv:meta): @pollute instead of @keep.
In particular, I want it to handle absolute import paths. It does
this for the return map; I forgot to add it here.
* src/current/compiler/map.xsl (lvmc:compile) [lvm:program-map]:
Preprocess nodes.
A classification used to be generated for each param for convenience;
this has been deprecated by the progui package generation (when `q:'
is used).
* src/current/include/preproc/macros.xsl: Remove template.
And everything else.
This is a big (important) change; it addresses one of the greatest
pains of the system.
Keeps were added during the DSL rewrite (to support symbols and such)
to work around the issue that there was no symbol-driven map; it
allowed symbols to persist disjoint from the `__yield' dependency
graph so that they could be mapped back and used by external systems.
The problem with that is that it's both messy (coupling the concept of
external dependencies with the actual code) and difficult to work
with. It had a huge performance impact on the linker for two reasons:
- Checking whether a package had already been seen and importing the
keeps on first visit was expensive because of tree searching and
manipulation; and
- _every_ keep was imported and processed by the linker, even if it
wouldn't end up being used by a particular program.
The later especially had huge performance impacts on the entire
system.
The entire dependency graph is now map-driven, with the exception of
the implicit `__yield' (which will eventually be moved into the map as
well and the magic `lv:yield' removed in favor of a template).
Performance-wise: our largest program ("dwelling") has many thousands
of symbols and the largest package imported the majority of them, many
of them unneeded, as the result of @keep subgraphs. Compilation of
the largest package within that (for the UI) took about a minute and a
half and ate up ~6GiB of RAM, for what really is a trivial task of
resolving externs, some basic symbol processing, a topological sort,
and ordering code fragments.
After this change, it takes ~15s and less than 2GiB of RAM. Still a
lot---and more improvements can be made---but much, much better.
@keep and friends was left in rater.xsd so that nothing breaks while
code is cleaned up; it'll be removed in the future.
* src/current/compiler/linker.xsl: Remove @keep support.
* src/current/dot/attr-keep.xsl: Remove now-unneeded template.
* src/current/dot/defnode.xsl: Remove @keep and related.
* src/current/include/preproc/eligclass.xsl: Remove @keep and related.
* src/current/include/preproc/expand.xsl: Remove @keep and related.
* src/current/include/preproc/macros.xsl: Remove @keep and related.
* src/current/include/preproc/symtable.xsl: Remove @keep and related.
* src/current/rater.xsd: Add TODO to remove @keep and friends.
This solves a long-standing problem whereby relative paths are not
properly resolved, leading to incorrect symbol conflicts when relative
paths to the same package vary between two imports.
See doc/notes/path-processing for an illustration.
* src/current/include/preproc/path.xsl
(preproc:resolve-relative-import): Add function.
(preproc:resolv-path, preproc:resolve-path, preproc:get-path): Add
functions to invoke existing templates.
* src/current/include/preproc/symtable.xsl (preproc:symimport): Use
`preproc:resolve-relative-import'.
* doc/notes/path-processing: Add notes.
This is a backwards-incompatible change that, like the input map,
requires the use of symbols in the return map. This will allow us to
forego the use of @keep and will have the return map be the authority
of what gets linked (all of its dependencies).
* src/current/compiler/map.xsl: Add symbol support to return-map.
This allows for the proper importing of symbols into the package
generated by the map compiler, which in turn allows for processing
their default values.
`set_defaults' wasn't in scope of maps.
* src/current/compiler/js.xsl (compiler:exit-rater lv:package):
Remove static output.
* src/current/compiler/linker.xsl (l:link-deps lv:package):
Link static after all other blocks, at highest scope within the
compiled module.
The `preproc:param-dim' template must remain for now, as it is used by
others.
* src/current/include/preproc/symtable.xsl
(preproc:symtable)[lv:param]: Extracted template.
* src/symtable/symbols.xsl
(preproc:symtable)[lv:param]: Added template.
* test/symtable/symbols.xsl
(lv:param): Scenario added.
It's important that others understand the system before I keep adding to
the mystery. This is at least a good start.
It also obviates certain awkward design issues that have evolved over
time and need addressing. Note that it also mentions that `keep' is
marked for removal---this is the situation that prompted this
documentation; changes will be made to work toward its removal to
improve the terrible linker performance when given many thousands of
symbols defined in over 500 separate packages.
* src/symtable.xsl: Add menu for symbols.
* src/symtable/symbols.xsl: Added.
This allows for nice formatting with newlines (to avoid excessively long
lines) without affecting the output.
* src/current/include/preproc/expand.xsl
(preproc:expand lv:classify/@desc): Normalize lv:classify/@desc whitespace
(Copyright headers will be added in the next commit; these are the
original files, unaltered in any way.)
The internal project name at LoVullo is simply "Calc DSL". This
liberates the entire thing. If anything was missed, I'll be added
later.
To continue building at LoVullo with this move, symlinks are used for
the transition; this is the exact code that is used in production.
There is a lot here---over 25,000 lines. Much of it is in disarray from
the environment surrounding its development, but it does work well for
what it was intended to do.
(LoVullo folks: fork point is 65723a0 in calcdsl.git.)
The thrush reference was also removed, as TAME does not currently make
use of it.
* .gitignore: Update path accordingly.
* configure.ac: Update path accordingly.
* src/init.xsl.in: Moved from conf.xsl.in.
This makes changes necessary to new ref format (which includes arity).
* src/graph.xsl: @ref function arity added
* src/preproc/expand/expand-sequence.xsl: @ref function arity added
* tools/literate-xsl: Submodule updated
This was already happening, but formally test and document it, since
it's a very useful property.
* src/graph.xsl (graph:union): Document dedupe behavior
* test/graph-test.xsl: Add test data
* test/graph.xspec: Add dedupe test for graph:union
Mike Gerwitz
Mike Gerwitz