This is a major step toward normalcy---removing the kluge of a build process
that was causing so many issues. Rather than echoing all operations to a
queue file before passing it off to dslc, the new build scripts in `bin/'
are used to invoke tame normally, as needed. This solves all of the current
issues with things not rebuilding when they should. And, as a bonus, tab
completion on targets works.
Sorry this took so long. There wasn't much motivation until we hired so
many people that are suffering from this.
This does a few major things, along with some miscellaneous others:
- Invoke bin/tame directly;
- Merge Makefile.2.in into Makefile.am; and
- Fix up some targets.
* build-aux/Makefile.2.in: Delete file. Mostly merged with Makefile.am.
* build-aux/Makefile.am: Add a bunch of new targets and definitions from
Makefile.2.in. Modify all that previously used .cqueue to now invoke
`$(TAME)' directly. Remove miscellaneous targets for trying to proxy
targets to Makefile.2.
(saneout, _go): Remove definitions.
(.NOTPARALLEL): Add to prevent parallel builds.
(ui/program.expanded.xml)[.version.xml]: Remove dependency for now.
(clean): Also clean generated PHP files. Follow symlinks to clean core.
This is still incomplete (does not clean all rate table stuff).
(suppliers.mk)[xmlo_cmd]: Remove. See `gen-make' and `gen-c1make'.
(lvroot)[summary-html]: New dependency.
(kill-tamed, tamed-die): New targets (former alias of latter) to kill
tamed.
* build-aux/gen-c1make: Generate `$(TAME)' invocation.
* build-aux/gen-make: Likewise. Remove `xmlo_cmd' output. Ignore recursive
`tame' symlink (this can be removed once we clean `rater/' up.
* build-aux/m4/calcdsl.m4 (TAME): Update description to reflect that it
should now be the path to `bin/tame'. Adjust `AC_CHECK_FILE' lines
accordingly.
(tame_needed_ver): Remove. We have been in the same repo as TAME itself
for quite some time. Remove associated code.
(AC_CONFIG_FILES): Remove `Makefile.2'.
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
Perform validation prefore `compile' command rather than a separate
`validate' step. Remove `rm'.
[compileSrc]: Stop echoing command. This was only necessary because of
the previous Makefile klugery; now Make echoes on its own correctly.
These scripts allow the TAME compiler stack to be invoked naturally, rather
than requiring the use of a Makefile today. This will not only allow users
to more easily invoke the compiler, but will also allow us to invoke TAME
naturally from Makefile and remove the klugery that has existed for so
long.
This users a server/client architecture in order to mitigate the startup
cost of the JVM. More documentation will follow.
Note that there are a bunch of symlinks in rater/---this is a transition
step to allow the build to continue working as it did before, which relies
on a directory structure that exists outside of this repository. This will
be cleaned up in the future.
* .gitignore (bin/dslc): Add ignore for generated file.
* bin/dslc.in: New script to encapsulate Java invocation.
* bin/tame: New script (client).
* bin/tamed: New script (server).
* configure.ac (JAVA_OPTS, DSLC_CLASSPATH, AUTOGENERATED): New variables for
dslc.in. Output bin/dslc.
* rater/README.md: Note that this symlink mess is temporary.
* rater/c1map: New symlink for dslc assumptions.
* rater/c1map.xsl: Likewise.
* rater/calc.xsd: Likewise.
* rater/compile.xsl: Likewise.
* rater/compiler: Likewise.
* rater/dot.xsl: Likewise.
* rater/include: Likewise.
* rater/link.xsl: Likewise.
* rater/standalone.xsl: Likewise.
* rater/summary.xsl: Likewise.
* rater/tame: Likewise (warning: circular symlink).
* src/current/src/com/lovullo/dslc/DslCompiler.java (_DslCompiler)[compile]:
Output `DONE' lines.
This will now automatically build on recursive target `all'.
* Makefile.am (SUBDIRS): Add `src/current/src'.
* src/current/src/Makefile: (.PHONY): Add `all'.
(all): New target. Alias to `dslc'.
This was throwing a warning in non-ancient versions of Saxon. It does not
need to be there, nor should it be, nor do I know why it was put there.
* src/current/include/preproc/template.xsl (eseq:is-expandable): Remove
@override.
This has a significant performance impact: processing time is cut in about
half and memory usage is reduced by more than 50%. For example, a
package that previously took 30s and 2.1GiB of memory to link now takes
14s and less than 900MiB of memory.
I had tried to perform this optimization a couple years ago but was
thwarted (I think) by the classifier markers. The previous commit did away
with those. I'm encouraged by the gains from the low-hanging fruit.
* src/current/compiler/linker.xsl
(l:process-empty, l:stack-empty): Convert from l:pstack and
l:sym-stack (respectively) to empty preproc:sym sequences.
(l:depgen-process-sym)[preproc:sym]: Append to sequence rather than
outputting new l:sym-stack tree.
Update all annotations and uses accordingly.
This is something that I thought would be useful back in the day when TAME
was in its infancy, but it is not important. Rather than having the linker
spend time trying to figure out what symbols belong in the classifier---and
rather than keeping that complexity around---this simplifies things by
making the existing `classify' method simply perform _all_ calculations, and
then yield only the classification portion of the result.
This isn't a problem in practice because, if we only desire the use of a
classifier, then we create a "supplier" that only uses classifications and
has no other dependencies. The end result is, as far as we care, the same.
* src/current/compiler/js.xsl (compiler:entry-rater)[lv:package]: Initialize
`classes' rather than invoking classifier
(compiler:entry-classifier)[lv:package]: Invoke all calculations and
return only classes to provide equivalent behavior.
(compiler:exit-classifier): Post-process classifications from calculation
results, iterating through classmap.
(compiler:classifier-yields-map)[lv:package]: Output all classifications
that are not generated. This differs slightly from the original
implementation in that it includes all non-generated classes rather than
just classes that have a non-generated `@yields'; this distinction is
important since `compiler:exit-classifier' is now using it to produce a
classification result set that doesn't contain all the generated
stuff (since it didn't before, and shouldn't now).
* src/current/compiler/linker.xsl: Update copyright year.
(l:resolv-deps)[preproc:sym[@l:mark-inclass]]: Remove template.
(l:resolv-deps)[preproc:sym...@l:mark-inclass...]: Remove template.
(l:depgen-sym): Set type of result to `element(preproc:sym)', since
`l:mark-inclass' is no longer produced.
[inclass, needs-class-mark]: Remove variables and all instances where
they are used.
(l:dep-aug)[inclass]: Remove param. Stop producing `@inclass' attribute.
(l:link-classifier)[lv:package]: Do not process any dependencies. This
can be removed entirely in the future since it now only produces static
code, which we can perhaps combine with a different block.
(l:link-rater)[lv:package]: Remove mention of `inclass' for dependencies;
all dependencies will now be compiled into this block.
This includes a SHA256 implementation which is _not_ intended for secure
cryptographic operations; see src/js/sha256.js header for more information.
* src/current/compiler/js.xsl (compiler:static): Echo src/js/sha256.js.
[map_method_uppercase, map_method_hash]: New functions.
* src/current/link.xsl: Include dslc-base.xsl.
* src/js/sha256.js: New file.
* src/current/compiler/map.xsl
(lvmc:get-method-func, lvmc:value-ref, lvmc:transformation-wrap): New
functions, partyl extracted from below.
(lvmc:compile)[lvm:map//lvm:from]: Use `lvmc:value-ref'.
[lvm:map//lvm:from/lvm:translate]: Add `[@key]' to match.
[lvm:map//lvm:transform]: New match. Ignore node entirely.
(lvmc:concat-compile): Propagate symtable to `lvmc:compile'.
* src/current/include/dslc-base.xsl (__path-root): New param.
* src/current/src/com/lovullo/dslc/DslCompiler.java
(DslCompiler)[compile]: Resolve TAME root path.
[_transform]: Set it.
DEV-3115
This is all really confusing because this doesn't use the same import
specification as packages; maps got stuck in a partial transition. So,
let's provide some helpful errors rather than silently failing.
* src/current/compiler/map.xsl (preproc:symtable)[lvm:import]:
Error if missing `@path'. Provide more information if `@package' was
provided to help clarify.
We need to cut down on symbol imports as much as possible; the whole system
starts dragging if we are importing thousands of symbols into a single
package.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate,c:*]: Mark
as local if `@preproc:generated`.
* src/current/include/preproc/template.xsl (preproc:macros)[lv:inline-template]:
Mark generated templates as such.
* src/symtable/symbols.xsl (preproc:symtable)[lv:template]: Mark as local if
`@preproc:generated'.
* src/current/compiler/js.xsl (compiler:js-number): New function to
remove leading zeroes.
(compile)[lv:const]: Use it.
* src/current/compiler/js-calc.xsl (compile-calc)[c:const]: Use it.
* src/current/compiler/js.xsl (compile-class-condtion)[lv:rate]: Do not
consider @no's in predicate generation when `@preproc:gentle-no' is set.
* src/current/include/preproc/macros.xsl (preproc:macros)[lv:rate-each]: Set
`@preproc:gentle-no' on generated `lv:rate', since the generator handles
`@no' itself.
* src/current/include/preproc/template.xsl
(preproc:gen-param-value)[lv:param-sym-value]: Suppress warning for
missing symbol and yield empty string if `@ignore-missing='true'`.
This ensures that they are compiled into the `consts' object.
* src/current/include/depgen.xsl (preproc:depgen)[lv:typedef]: Include
`lv:enum/lv:item/@name' as dependencies.
The problem with this implementation was that, any time a generator had an
associated generated @yields (which is common), it wouldn't be included in
the summary page.
We can address this in the future. It's not necessarily that it was
incorrect; it's just how the system made use of it.
* src/current/include/preproc/symtable.xsl (preproc:symtable)[lv:rate]:
Do not mark @preproc:yields-generated symbols as @preproc:generated.
Templates can expand into unexpected places, so sometimes warnings are
inappropriately issued.
* src/current/include/depgen.xsl (preproc:depgen)[lv:template]: Ignore.
[lv:template/lv:param]: Remove (now unnecessary with above).
In order for the cmatch algorithm to work properly, predicates must be
re-ordered on @dim descending.
* src/current/compiler/js.xsl (compile)[lv:classify]: Order all different
dimensions, not just scalars.