employer
/
tame
1
0
Fork 0
Code Packages Releases Activity

depgen: Quadratic=>linear-time algorithm 

This is a significant performance improvement for dependency
generation (which is responsible for building the dependency graph for a
package).

The previous algorithm ran in O(n²) time: it would iterate over the given
symbol table, and for _each_ symbol, do a linear scan of the entire document
to search for the corresponding source block.  This resulted in explosive
depgen time for larger packages.

This makes the algorithm run in O(n) by:
  - Using an XSLT 3 map for the symbol table for O(1) lookups; and
  - Iterating over the _document_ a single time rather than the symbol
    table, referencing the symbol table as needed (in O(1) time).

There are other parts of the system that can benefit from these same
improvements.  This is important, since we need to be able to handle many
thousands of symbols efficiently.

* src/current/compiler/linker.xsl (l:depgen-sym): Recognize smybol `no-deps'
    property, permitting missing dependencies.  This allows us to avoid
    creating nonsense nodes just to satisfy the linker, while still allowing
    the linker to perform essential checks to defend against compiler bugs.
* src/current/compiler/map.xsl (lvmc:stub-symtable): Set @no-deps on
    `___head' and `___tail' symbols.
  (lvmc:mapsym): Set `no-deps' as appropriate on map symbols.
  (preproc:depgen)[lvm:map[@from]]: Generate `preproc:sym-dep' node, which
    is now expected by the depgen process.
  (preproc:depgen)[lvm:map[*]]: Likewise.
  (preproc:depgen)[*[@lvmc:type='retmap']//lvmm:map[@from]]: Remove
    unnecessary template.
  (preproc:symtable)[lvm:map[@value]]: Pass `no-deps' to `lvmc:mapsym'.
* src/current/include/depgen.xsl (preproc:depgen)[preproc:symtable]: Create
    and use XSLT 3 map in place of `preproc:symtable' tree.  This allows for
    constant-time lookups.  Provide to templates via tunnelling.  Use it in
    place of exiting tree references.  Process source tree rather than
    iterating over symbol table.
  (preproc:depgen)[lv:rate, c:sum[@generates], c:product[@generates],
    lv:classify, lv:function/lv:param, lv:function, lv:typedef]: Produce
      `preproc:sym-dep' nodes (which was previously done while iterating
      over the symbol table).
  (preproc:depgen)[preproc:sym]: Remove all such processing, since we no
    longer iterate over the symbol table.
  (preproc:depgen)[c:value-of]: Use symtable map.
  (preproc:depgen-match): Likewise.
  (preproc:depgen)[lv:union]: Modify to handle changes to lv:typedef
    template.
  (preproc:depgen)[text()]: Remove and replace with `node()'.
* src/current/include/preproc/package.xsl (preproc:resolv-syms): Remove
    logging of symbol resolution.  This has a slight performace impact since
    there is a lot of output.
* src/current/include/preproc/symtable.xsl
  (lv:function/lv:param, c:let/c:Values/c:value): Set `no-deps'.
* src/symtable/symbols.xsl: Add documentation of `no-deps'.
  (preproc:symtable)[lv:meta]: Set `no-deps'.
master
Mike Gerwitz 2019-02-07 11:12:27 -05:00
parent 8e7a946127
commit b6cfdb4221
6 changed files with 163 additions and 155 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
src/current/compiler/linker.xsl
View File

@ -673,15 +673,20 @@
$l:orig-root/lv:package
" />
<variable name="name" as="xs:string"
select="@name" />
<variable name="name" as="xs:string" select="@name" />
<variable name="no-deps" as="xs:boolean"
select="if ( @no-deps = 'true' ) then true() else false()" />
<variable name="deps" as="element( preproc:sym-dep )?"
select="$pkg/preproc:sym-deps
/preproc:sym-dep[ @name=$name ]" />
select="if ( $no-deps ) then
()
else
$pkg/preproc:sym-deps
/preproc:sym-dep[ @name=$name ]" />
<!-- if we could not locate the dependencies, then consider this to be an
error (even if there are no deps, there should still be a list dfn) -->
<if test="not( $deps )">
<if test="not( $no-deps or $deps )">
<call-template name="log:internal-error">
<with-param name="name" select="'link'" />
<with-param name="msg">

54
src/current/compiler/map.xsl
View File

@ -2,7 +2,7 @@
<!--
Compiles map fragments to produce a map from source data to a destination
Copyright (C) 2016, 2018 R-T Specialty, LLC.
Copyright (C) 2016, 2018, 2019 R-T Specialty, LLC.
This file is part of TAME.
@ -237,10 +237,12 @@
<preproc:sym name=":{$type-prefix}:___head"
type="{$type-prefix}:head"
pollute="true"
ignore-dup="true" />
ignore-dup="true"
no-deps="true" />
<preproc:sym name=":{$type-prefix}:___tail"
type="{$type-prefix}:tail"
ignore-dup="true" />
ignore-dup="true"
no-deps="true" />
</preproc:symtable>
</template>
@ -249,6 +251,7 @@
<param name="name" />
<param name="from" />
<param name="type-prefix" select="/lv:package/@lvmc:type" />
<param name="no-deps" as="xs:boolean" select="false()" />
<!-- allow mappings to be overridden after import, which allows defaults
to be set and then overridden -->
@ -266,6 +269,10 @@
<if test="$from">
<copy-of select="$from" />
</if>
<if test="$no-deps">
<attribute name="no-deps" select="'true'" />
</if>
</preproc:sym>
</template>
@ -333,8 +340,18 @@
</call-template>
</template>
<template match="lvm:pass" mode="preproc:depgen" priority="5">
<preproc:sym-ref name="{@name}" lax="true" />
<preproc:sym-dep name=":map:{@name}">
<preproc:sym-ref name="{@name}" lax="true" />
</preproc:sym-dep>
</template>
<template match="lvm:pass[ root(.)/@lvmc:type = 'retmap' ]"
mode="preproc:depgen" priority="6">
<preproc:sym-dep name=":retmap:{@name}">
<preproc:sym-ref name="{@name}" lax="true" />
</preproc:sym-dep>
</template>
@ -385,15 +402,17 @@
<template match="lvm:map[ @from
and root(.)/@lvmc:type = 'map' ]"
mode="preproc:depgen" priority="5">
<!-- to the DSL -->
<preproc:sym-ref name="{@to}" lax="true" />
<preproc:sym-dep name=":map:{@to}">
<preproc:sym-ref name="{@to}" lax="true" />
</preproc:sym-dep>
</template>
<template match="lvm:map[ @from
and root(.)/@lvmc:type = 'retmap' ]"
mode="preproc:depgen" priority="5">
<!-- from the DSL -->
<preproc:sym-ref name="{@from}" lax="true" />
mode="preproc:depgen" priority="6">
<preproc:sym-dep name=":retmap:{@to}">
<preproc:sym-ref name="{@from}" lax="true" />
</preproc:sym-dep>
</template>
<template match="lvm:map[ @from ]" mode="preproc:depgen" priority="4">
@ -401,10 +420,6 @@
select="'internal error: unhandled lvm:map: ', ." />
</template>
<template match="/*[ @lvmc:type='retmap' ]//lvm:map[ @from ]" mode="preproc:depgen" priority="6">
<preproc:sym-ref name="{@from}" lax="true" />
</template>
<!--
Triggers dependency generation on the source document, which contains far more
@ -560,6 +575,7 @@
<template match="lvm:map[ @value ]" mode="preproc:symtable" priority="5">
<call-template name="lvmc:mapsym">
<with-param name="name" select="@to" />
<with-param name="no-deps" select="true()" />
</call-template>
</template>
@ -612,15 +628,19 @@
</template>
<template match="lvm:map[ * ]" mode="preproc:depgen" priority="5">
<preproc:sym-ref name="{@to}" lax="true" />
<preproc:sym-dep name=":map:{@to}">
<preproc:sym-ref name="{@to}" lax="true" />
</preproc:sym-dep>
</template>
<template match="lvm:map[ *
and root(.)/@lvmc:type = 'retmap' ]"
mode="preproc:depgen" priority="6">
<for-each select=".//lvm:from">
<preproc:sym-ref name="{@name}" lax="true" />
</for-each>
<preproc:sym-dep name=":retmap:{@to}">
<for-each select=".//lvm:from">
<preproc:sym-ref name="{@name}" lax="true" />
</for-each>
</preproc:sym-dep>
</template>

223
src/current/include/depgen.xsl
View File

@ -2,7 +2,7 @@
<!--
Dependency generation
Copyright (C) 2016, 2018 R-T Specialty, LLC.
Copyright (C) 2016, 2018, 2019 R-T Specialty, LLC.
This file is part of TAME.
@ -28,9 +28,11 @@
<xsl:stylesheet version="1.0"
xmlns="http://www.w3.org/1999/xhtml"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:map="http://www.w3.org/2005/xpath-functions/map"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:preproc="http://www.lovullo.com/rater/preproc"
xmlns:lv="http://www.lovullo.com/rater"
xmlns:lvm="http://www.lovullo.com/rater/map"
xmlns:t="http://www.lovullo.com/rater/apply-template"
xmlns:c="http://www.lovullo.com/calc"
xmlns:ext="http://www.lovullo.com/ext"
@ -75,22 +77,41 @@
<xsl:template match="preproc:symtable" mode="preproc:depgen" priority="9">
<xsl:variable name="symtable" select="." />
<preproc:sym-deps>
<!-- process dependencies for all non-imported symbols -->
<xsl:for-each select="preproc:sym[ not( @src ) ]">
<xsl:variable name="cursym" select="." />
<xsl:variable name="symtable-map" as="map( xs:string, element( preproc:sym ) )"
select="map:merge( for $sym in $symtable/preproc:sym
return map{ string( $sym/@name ) : $sym } )" />
<xsl:variable name="deps">
<preproc:deps>
<xsl:apply-templates select="." mode="preproc:depgen" />
</preproc:deps>
</xsl:variable>
<preproc:sym-deps>
<xsl:variable name="deps" as="element( preproc:sym-dep )*">
<xsl:apply-templates mode="preproc:depgen"
select="root(.)/lv:*,
root(.)/lvm:*">
<xsl:with-param name="symtable-map" select="$symtable-map"
tunnel="yes" />
</xsl:apply-templates>
</xsl:variable>
<!-- preproc:sym-deps may be nested -->
<xsl:for-each select="$deps, $deps//preproc:sym-dep">
<xsl:variable name="sym-name" as="xs:string"
select="@name" />
<xsl:variable name="cursym" as="element( preproc:sym )?"
select="$symtable-map( $sym-name )" />
<xsl:if test="not( $cursym )">
<xsl:message select="." />
<xsl:message terminate="yes"
select="concat( 'internal error: ',
'cannot find symbol in symbol table: ',
'`', $sym-name, '''' )" />
</xsl:if>
<!-- do not output duplicates (we used to not output references
to ourselves, but we are now retaining them, since those
data are useful) -->
<xsl:variable name="uniq" select="
$deps/preproc:deps/preproc:sym-ref[
preproc:sym-ref[
not( @name=preceding-sibling::preproc:sym-ref/@name )
]
" />
@ -111,9 +132,8 @@
<xsl:variable name="syms-rtf">
<xsl:for-each select="$uniq">
<xsl:variable name="name" select="@name" />
<xsl:variable name="sym" select="
$symtable/preproc:sym[ @name=$name ]
" />
<xsl:variable name="sym" as="element( preproc:sym )?"
select="$symtable-map( $name )" />
<!-- we should never have this problem. -->
<xsl:if test="not( $sym ) and not( @lax='true' )">
@ -144,6 +164,7 @@
</preproc:sym>
</xsl:for-each>
</xsl:variable>
<xsl:variable name="syms" select="$syms-rtf/preproc:sym" />
<!-- only applicable if the symbol is @lax and the symbol was not
@ -259,102 +280,72 @@
</xsl:template>
<xsl:template match="preproc:sym[ @extern='true' ]" mode="preproc:depgen" priority="9">
<!-- externs will be processed once they are resolved in another package -->
<xsl:template mode="preproc:depgen" priority="7"
match="lv:rate">
<preproc:sym-dep name="{@yields}">
<xsl:apply-templates mode="preproc:depgen" />
</preproc:sym-dep>
</xsl:template>
<!-- all symbols with a @parent (e.g. generators) should depend on the parent
itself (which of course introduces the parent's dependencies into the tree) -->
<xsl:template match="preproc:sym[ @parent ]" mode="preproc:depgen" priority="7">
<preproc:sym-ref name="{@parent}" />
<xsl:template mode="preproc:depgen" priority="7"
match="c:sum[@generates]
|c:product[@generates]">
<preproc:sym-dep name="{@generates}">
<preproc:sym-ref name="{ancestor::lv:rate[1]/@yields}" />
</preproc:sym-dep>
<!-- this may have other rules; see below -->
<xsl:next-match />
</xsl:template>
<xsl:template match="preproc:sym[ @type='rate' ]" mode="preproc:depgen" priority="5">
<xsl:variable name="name" select="@name" />
<xsl:variable name="pkg" as="element( lv:package )"
select="root(.)" />
<xsl:template mode="preproc:depgen" priority="7"
match="lv:classify">
<preproc:sym-dep name=":class:{@as}">
<xsl:apply-templates mode="preproc:depgen" />
</preproc:sym-dep>
<xsl:variable name="rate" as="element( lv:rate )*"
select="$pkg/lv:rate[ @yields=$name ]" />
<xsl:if test="count( $rate ) gt 1">
<xsl:message terminate="yes"
select="'error: rate block name conflict:',
string( $rate/@yields[1] )" />
<xsl:if test="@yields">
<preproc:sym-dep name="{@yields}">
<preproc:sym-ref name=":class:{@as}" />
</preproc:sym-dep>
</xsl:if>
<xsl:apply-templates mode="preproc:depgen"
select="$rate" />
</xsl:template>
<xsl:template match="preproc:sym[ @type='class' ]" mode="preproc:depgen" priority="5">
<!-- all class symbol names are prefixed with ":class:" -->
<xsl:variable name="as" select="substring-after( @name, ':class:' )" />
<xsl:variable name="pkg" as="element( lv:package )"
select="root(.)" />
<xsl:template mode="preproc:depgen" priority="8"
match="lv:function/lv:param">
<xsl:variable name="fname" as="xs:string"
select="parent::lv:function/@name" />
<xsl:apply-templates
select="$pkg/lv:classify[ @as=$as ]"
mode="preproc:depgen" />
<preproc:sym-dep name=":{$fname}:{@name}">
<preproc:sym-ref name="{$fname}" />
</preproc:sym-dep>
</xsl:template>
<xsl:template mode="preproc:depgen" priority="7"
match="lv:param">
<preproc:sym-dep name="{@name}">
<preproc:sym-ref name="{@type}" />
<xsl:apply-templates mode="preproc:depgen" />
</preproc:sym-dep>
</xsl:template>
<xsl:template match="preproc:sym[ @type='param' ]" mode="preproc:depgen" priority="5">
<xsl:variable name="name" select="@name" />
<xsl:apply-templates
select="root(.)/lv:param[ @name=$name ]"
mode="preproc:depgen" />
<xsl:template mode="preproc:depgen" priority="7"
match="lv:function">
<preproc:sym-dep name="{@name}">
<xsl:apply-templates mode="preproc:depgen" />
</preproc:sym-dep>
</xsl:template>
<xsl:template match="preproc:sym[ @type='func' ]" mode="preproc:depgen" priority="5">
<xsl:variable name="name" select="@name" />
<xsl:variable name="pkg" as="element( lv:package )"
select="root(.)" />
<xsl:apply-templates
select="$pkg/lv:function[ @name=$name ]"
mode="preproc:depgen" />
</xsl:template>
<xsl:template match="preproc:sym[ @type='type' ]" mode="preproc:depgen" priority="5">
<xsl:variable name="name" select="@name" />
<xsl:variable name="pkg" as="element( lv:package )"
select="root(.)" />
<!-- a typedef could optionally be contained within another typedef -->
<xsl:apply-templates mode="preproc:depgen" select="
$pkg/lv:typedef[ @name=$name ]
, $pkg/lv:typedef//lv:typedef[ @name=$name ]
" />
</xsl:template>
<xsl:template match="preproc:sym[ @type='lparam' ]" mode="preproc:depgen" priority="5">
<!-- do nothing -->
</xsl:template>
<xsl:template match="preproc:sym[ @type='const' ]" mode="preproc:depgen" priority="5">
<!-- do nothing -->
</xsl:template>
<xsl:template match="preproc:sym[ @type='tpl' ]" mode="preproc:depgen" priority="5">
<!-- do nothing -->
</xsl:template>
<xsl:template match="preproc:sym[ @type='meta' ]" mode="preproc:depgen" priority="5">
<!-- do nothing -->
</xsl:template>
<xsl:template match="preproc:sym" mode="preproc:depgen" priority="1">
<xsl:message terminate="yes">
<xsl:text>[depgen] error: unexpected symbol </xsl:text>
<xsl:sequence select="." />
</xsl:message>
<xsl:template mode="preproc:depgen" priority="7"
match="lv:typedef">
<preproc:sym-dep name="{@name}">
<xsl:apply-templates mode="preproc:depgen" />
</preproc:sym-dep>
</xsl:template>
@ -363,14 +354,17 @@
<!-- ignore symbols within templates -->
</xsl:template>
<xsl:template name="preproc:depgen-c-normal" match="c:value-of|c:when" mode="preproc:depgen" priority="5">
<xsl:param name="name" select="@name" />
<xsl:param name="symtable-map" as="map(*)" tunnel="yes" />
<xsl:variable name="pkg" as="element( lv:package )"
select="root(.)" />
<xsl:variable name="sym"
select="$pkg/preproc:symtable/preproc:sym[ @name=$name ]" />
<xsl:variable name="sym" as="element( preproc:sym )?"
select="$symtable-map( $name )" />
<!-- see if there is a c:let associated with this name -->
<xsl:variable name="let" select="
@ -445,18 +439,13 @@
</xsl:template>
<xsl:template match="c:let/c:values/c:value" mode="preproc:depgen" priority="5">
<!-- do not consider the c:value name -->
<xsl:apply-templates mode="preproc:depgen" />
</xsl:template>
<xsl:template name="preproc:depgen-match">
<xsl:param name="on" select="@on" />
<xsl:param name="symtable-map" as="map(*)" tunnel="yes" />
<xsl:variable name="class" select="ancestor::lv:classify" />
<xsl:variable name="sym"
select="root(.)/preproc:symtable/preproc:sym[ @name=$on ]" />
<xsl:variable name="sym" as="element( preproc:sym )"
select="$symtable-map( $on )" />
<!-- are we depending on another classification? -->
<xsl:if test="$sym/@type='cgen'">
@ -524,23 +513,18 @@
</xsl:template>
<xsl:template match="lv:param" mode="preproc:depgen" priority="5">
<!-- while the type is reduced to a primitive, let's still include the
dependency symbol -->
<preproc:sym-ref name="{@type}" />
<xsl:template match="lv:union" mode="preproc:depgen" priority="5">
<xsl:for-each select="lv:typedef">
<preproc:sym-ref name="{@name}" />
</xsl:for-each>
<!-- we still need to process the typedefs independently -->
<xsl:apply-templates mode="preproc:depgen" />
</xsl:template>
<xsl:template match="lv:typedef" mode="preproc:depgen" priority="5">
<!-- we depend on any types that we create a union of -->
<xsl:for-each select="lv:union/lv:typedef">
<preproc:sym-ref name="{@name}" />
</xsl:for-each>
<!-- and each of the constants generated by our type -->
<xsl:for-each select="lv:enum/lv:item">
<preproc:sym-ref name="{@name}" />
</xsl:for-each>
<xsl:template match="lv:enum/lv:item" mode="preproc:depgen" priority="5">
<preproc:sym-ref name="{@name}" />
</xsl:template>
@ -555,9 +539,8 @@
<xsl:apply-templates mode="preproc:depgen" />
</xsl:template>
<xsl:template match="text()" mode="preproc:depgen" priority="2">
<!-- not interested. nope. -->
<xsl:template match="node()" mode="preproc:depgen" priority="1">
<!-- skip -->
</xsl:template>
</xsl:stylesheet>

9
src/current/include/preproc/package.xsl
View File

@ -739,15 +739,6 @@
</xsl:message>
</xsl:if>
<!-- logging -->
<xsl:message>
<xsl:text>[preproc] resolved `</xsl:text>
<xsl:value-of select="$name" />
<xsl:text>' dimensions as `</xsl:text>
<xsl:value-of select="$max" />
<xsl:text>'</xsl:text>
</xsl:message>
<!-- copy, substituting calculated dimensions -->
<xsl:copy>
<xsl:sequence select="@*" />

5
src/current/include/preproc/symtable.xsl
View File

@ -861,7 +861,8 @@
<xsl:variable name="fname" select="parent::lv:function/@name" />
<preproc:sym name=":{$fname}:{@name}" parent="{$fname}" varname="{@name}"
type="lparam" dtype="{@type}" dim="{$dim}" desc="{@desc}" tex="{@sym}">
type="lparam" dtype="{@type}" dim="{$dim}" desc="{@desc}" tex="{@sym}"
no-deps="true">
<!-- may or may not be defined -->
<xsl:sequence select="@default" />
@ -889,7 +890,7 @@
as a symbol -->
<preproc:sym name=":{$lname}:{@name}" local="true" varname="{@name}"
type="lparam" dtype="{@type}" dim="{$dim}" desc="{@desc}" tex="{@sym}"
lparent="{$lname}" />
lparent="{$lname}" no-deps="true" />
<xsl:apply-templates mode="preproc:symtable" />
</xsl:template>

12
src/symtable/symbols.xsl
View File

@ -2,7 +2,7 @@
<!--
Semantic analysis for symbol generation
Copyright (C) 2016 R-T Specialty, LLC.
Copyright (C) 2016, 2019 R-T Specialty, LLC.
This file is part of TAME.
@ -98,6 +98,13 @@
is the result of generated code
(e.g. @ref{Macro Expansion,,template expansion}).
@item no-deps
When @code{true},
linker does not attempt to look up dependencies for this symbol.
Otherwise,
as a safeguard against compilation bugs,
the linker will fail if a symbol is missing a dependency list.
@item allow-circular
Permit@tie{}@sym{} to be part of a cycle (@pxref{Dependency Graph}).
This is desirable for recursive functions, but should otherwise be
@ -427,7 +434,8 @@
<preproc:sym type="meta"
name=":meta:{@name}"
desc="Metavalue {@name}"
pollute="true" />
pollute="true"
no-deps="true" />
</for-each>
</template>