tame/RELEASES.md

468 lines
17 KiB
Markdown
Raw Normal View History

TAME Release Notes
==================
This file contains notes for each release of TAME since v17.4.0.
TAME uses [semantic versioning][]. Any major version number increment
indicates that backwards-incompatible changes have been introduced in that
version. Each such version will be accompanied by notes that provide a
migration path to resolve incompatibilities.
2020-04-29 15:48:21 -04:00
[semantic versioning]: https://semver.org/
TAME developers: Add new changes under a "NEXT" heading as part of the
commits that introduce the changes. To make a new release, run
`tools/mkrelease`, which will handle updating the heading for you.
2021-03-18 09:55:36 -04:00
2022-03-07 12:24:31 -05:00
v19.0.2 (2022-03-07)
====================
This is a bugfix release that corrects applying param defaults via input
maps.
Compiler
--------
- Input maps using translations that fall back to `param/@default` will now
correctly apply that default.
- This was broken in the previous release v19.0.1.
2022-03-03 13:47:37 -05:00
v19.0.1 (2022-03-03)
====================
This is a bugfix release.
Compiler
--------
- Input maps will now ensure that non-numeric string values result in `0`
rather than `NaN`, the latter of which results in undefined behavior in
the new classification system.
2022-03-01 16:32:43 -05:00
v19.0.0 (2022-03-01)
====================
x/0=0 with global flag for new classification system This was originally my plan with the new classification system, but it was undone because I had hoped to punt on the somewhat controversial issue. Unfortunately, I see no other way. Here I attempt to summarize the reasons why, many of which are specific to the design decisions of TAME. Keep in mind that TAME is a domain-specific language (DSL) for writing insurance rating systems. It should act intuitively for our use case, while still being mathematically sound. If you still aren't convinced, please see the link at the bottom. Target Language Semantics (ECMAScript) -------------------------------------- First: let's establish what happens today. TAME compiles into ECMAScript, which uses IEEE 754-2008 floating-point arithmetic. Here we have: x/0 = Infinity, x > 0; x/0 = -Infinity, x < 0; 0/0 = NaN, x = 0. This is immediately problematic: TAME's calculations must produce concrete real numbers, always. NaN is not valid in its domain, and Infinity is of no practical use in our computational model (TAME is build for insurance rating systems, and one will never have infinite premium). Put plainly: the behavior is undefined in TAME when any of these values are yielded by an expression. Furthermore, we have _three different possible situations_ depending on whether the numerator is positive, negative, or zero. This makes it more difficult to reason about the behavior of the system, for values we do not want in the first place. We then have these issues in ECMAScript: Infinity * 0 = NaN. -Infinity * 0 = NaN. NaN * 0 = NaN. These are of particular concern because of how predicates work in TAME, which will be discussed further below. But it is also problematic because of how it propagates: once you have NaN, you'll always have NaN, unless you break out of the situation with some control structure that avoids using it in an expression at all. Let's now consider predicates: NaN > 0 = false. NaN < 0 = false. NaN === 0 = false. NaN === NaN = false. These will be discussed in terms of classification predicates (matches). We also have issues of serialization: JSON.stringify(Infinity) = "null". JSON.stringify(NaN) = "null". These means that these values are difficult to transfer between systems, even if we wanted them. TAME's Predicates ----------------- TAME has a classification system based on first-order logic, where ⊥ is represented by 0 and ⊤ is represented by 1. These classifications are used as predicates to calculations via the @class attribute of a rate block. For example: <rate-each class="property" generates="propValue" index="k"> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </rate> As can be observed via the Summary Page, this calculation compiles into the following mathematical expression: ∑ₖ(pₖ(tₖ/dₖ)), that is—the quotient is then multiplied by the value of the `property` classification, which is a 0 or 1 respectively for that index. Let's say that tivPropDivisor were defined in this way: <rate-each class="property" generates="tivPropDivisor" index="k"> <!--- ... logic here ... --> </rate> It does not matter what the logic here is. Observe that the predicate here is `property` as well, which means that, if this risk is not a property risk, then `tivPropDivisor` will be `0`. Looking back at `propValue`, let's say that we do have a property risk, and that `buildingTiv` is `[100_000, 200_000]` and `tivPropDivisor` is 1000. We then have: 1(100,000 / 1000) + 1(200,000 / 1000)) = 300. Consider instead what happens if `property` is 0. Since we have no property locations, we have `[0, 0]` as `buildingTiv` and `tivPropDivisor` is 0. 0(0/0) + 0(0/0)) = 0(NaN + NaN) = NaN. This is clearly not what was intended. The predicate is expected to be _strongly_ zero, as if using an Iverson bracket: ((0/0)[0] + (0/0)[0]) = 0. Of course, one option is to redefine TAME such that we use Iverson's convention in place of summation, however this is neither necessary nor desirable given that (a) NaN is not valid within the domain of any TAME expression, and (b) Summation is elegantly generalized and efficiently computed using vector arithmetic and SIMD functions. That is: there's no use in messing with TAME's computational model for a valid that should be impossible to represent. Short-Circuiting Computation ---------------------------- There's another way to look at it, though: that we intended to skip the computation entirely, and so it doesn't matter what the quotient is. If the compiler were smart enough (and maybe one day it will be), it would know that the predicate of `tivPropDivisor` and `propValue` are the same and so there is no circumstance under which we would compute `propValue` and have `tivPropDivisor` be 0. The problem is: that short-circuiting is employed as an _optimization_, and is an implementation detail. Mathematically, the expression is unchanged, and is still invalid within TAME's domain. It is unrepresentable, and so this is not an out. But let's pretend that it was defined that way, which would yield this: { ∑ₖ(pₖ(tₖ/dₖ)), ∀x∈p(x = 1); propValue = < { 0, otherwise. This is the optimization that is employed, but it's still not mathematically correct! What happens if p₀ = 1, but p₁ = 0? Then we have: 1(100,000/1000) + 0(0/0) = 100 + NaN = NaN, but the _intent_ was clearly to have 100 + 0 = 100, and so we return to the original problem once again. Classification Predicates and Intent ------------------------------------ Classifications are used as predicates for equations, but classifications _themselves_ have predicates in the form of _matches_. Consider, for example, a classification that may be used in an assertion to prevent negative premium from being generated: <t:assert failure="premBuilding must not be negative for any index"> <t:match-gte value="premBuilding" value="#0" /> </t:assert> Simple enough—the system will fail if the premium for a given building is below $0. But what happens if premBuilding is calculated as so? <rate-each class="property" yields="premBuildingTotal" generates="premBuilding" index="k"> <c:product> <c:value-of name="propValue" index="k" /> <c:value-of name="propRate" index="k" /> </c:product> </rate-each> Alas, if `property` is false for any index, then we know that `propValue` is NaN, and NaN * x = NaN, and so `premBuilding` is NaN. The above assertion will compile the match into the first-order sentence ∀x∈b(x > 0). Unfortunately, NaN is not greater than, less than, equal to, or any other sort of thing to 0, and so _this assertion will trigger_. This causes practical problems with the `_premium_` template, which has an `@allow-zero@` argument to permit zero premium. Consider this real-world case that I found (variables renamed), to avoid a strawman: <t:premium class="loc" round="cent" yields="locInitialTotal" generates="locInitial" index="k" allow-zero="true" desc="..."> <c:value-of name="premAdditional" /> <c:quotient> <c:value-of name="premLoc" index="k" /> <c:value-of name="premTotal" /> </c:quotient> </t:premium> This appears to be responsible for splitting up `premAdditional` relative to the total premium contribution of each location. It explicitly states that it wants to permit a zero value. The intent of this block is clear: a value of 0 is explicitly permitted and _expected_. But if `premTotal` is for whatever reason 0—whether it be due to a test case or some unexpected input—then it'll yield a NaN and make the entire expression NaN. Or if `premAdditional` or `premLoc` are tainted by a NaN, the same result will occur. The assertion will trigger. And, indeed, this is what I'm seeing with test cases against the new classification system. What about Infinity? Is it intuitive that, should `propValue` in the previous example be positive and `propRate` be 0, that we would, rather than producing a very small value, produce an infinitely large one? Does that match intuition? Remember, this system is a domain-specific language for _our_ purposes—it is not intended to be used to model infinities. For example, say we had this submission because the premium exceeds our authority to write with some carrier: <t:submit reason="Premium exceeds authority"> <t:match-gt name="premBuilding" value="#100k" /> </t:submit> If we had (100,000 / 0) = ∞, then this submit reason would trigger. Surely that was not intended, since we have `property` as a predicate and `propRate` with the same predicate, implying that the answer we _actually_ want is 0! In that case, what we _probably_ want to trigger is something like <rate yields="premFinal"> <t:maxreduce> <c:value-of name="premBuildingTotal" /> <c:value-of name="#500" /> </t:maxreduce> </rate>, in order to apply a minimum premium of $500. But if `premBuildingTotal` is Infinity, then you won't get that—you'll get Infinity, which is of course nonsense. And nevermind -Infinity. Why Wasn't This a Problem Before? --------------------------------- So why bring this up now? Why have we survived a decade without this? We haven't, really—these bugs have been hidden. But the old classification system covered them up; predicates would implicitly treat missing values as 0 by enclosing them in `(x||0)` in the compiled code. Observe this ECMAScript code: NaN || 0 = 0. Consequently, the old classification system absorbed bad values and treated them implicitly as 0. But that was a bug, and had to be removed; it meant that missing indexes in classifications would trigger predicates that were not intended to be triggered, if they matched against 0, or matched against a value less than some number larger than zero. (See `core/test/core/class` for examples.) The new classification system does not perform such defaulting. _But it also does not expect to receive values outside of its valid domain._ Consequently, _NaN and Infinity lead to undefined behavior_, and the current implementation causes the predicate to match (NaN < 0) and therefore fail. The reason for this is because that this implementation is intended to convey precisely the computation necessary for the classification system, as formally defined, so that it can be later optimized even further. Checking for values outside the domain not only should not be necessary, but it would prevent such future optimizations. Furthermore, parameters used to compile into (param||0), to account for missing values or empty strings. This changed somewhat recently with 5a816a4701211adf84d3f5e09b74c67076c47675, which pre-cast all inputs and allowed relaxing many of those casts since they were both wasteful and no longer necessary. Given that, for all practical purposes, 0/0=0 in the system <1yr ago. Infinity, of course, is a different story, since (Infinity||0)=Infinity; this one has always been a problem. Let's Just Fail --------------- Okay, so we cannot have a valid expression, so let's just fail. We could mean that in two different ways: 1. Fail at runtime if we divide by 0; or 2. Fail at compile-time if we _could_ divide by 0. Both of these have their own challenges. Let's dismiss #2 right off the bat for now, because until we have TAMER, that's not really feasible. We need something today. We will discuss that in the future. For #1—we cannot just throw an error and halt computation, because if the `canterm` flag passed into the system is `false`, then _computation must proceed and return all results_. Terminating classifications are checked after returning rather than throwing errors. Since we have to proceed with computation, then the computations have to be valid, and so we're left with the same problem again—we cannot have undefined behavior. One could argue that, okay, we have undefined behavior, but we're going to fail because of the assertion anyway! That's potentially defensible, but it is at the moment undesirable, because we get so many failures. And, relative to the section below, it's not clear to me what benefit we get from that behavior other than making things more difficult for ourselves. Furthermore, such an assertion would have to be defined for every calculation that performs a quotient, and would have to set some intermediate flag in the calculation which would then have to be checked for after-the-fact. This muddies the generated calculation, which causes problems for optimizations, because it requires peering into state of the calculation that may be hidden or optimized away. If we decide that calculations must be valid because we cannot fail, and we have to stick with the domain of calculations, then `x/0` must be _something_ within that domain. x/0=0 Makes Sense With the Current System ----------------------------------------- Let's take a step back. Consider a developer who is unaware that NaN/Infinity are permitted in the system—they just know that division by zero is a bad thing to do because that's what they learned, and they want to avoid it in their code. Consider that they started with this: <rate-each class="property" generates="propValue" index="k"> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </rate> They have inspected the output of `tivPropDivisor` and see that it is sometimes 0. They understand that `property` is a predicate for the calculation, and so reasonably think that they could do something like this: <classify as="nonzero-tiv-prop-divisor" ...> <t:match-ne on="tivPropDivisor" value="#0" /> </classify> and then change the rate-each to <rate-each class="property nonzero-tiv-prop-divisor" ...>. Except that, of course, we know that will have no effect, because a NaN is a NaN. This is not intuitive. So they'd have to do this: <rate-each class="property" generates="propValue" index="k"> <c:cases> <c:case> <t:when-ne name="tivPropDivisor" value="#0" /> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </c:case> <c:otherwise> <c:value-of name="#0" /> </c:otherwise> </c:cases> </rate>. But for what purpose? What have we gained over simply having x/0=0, which does this for you? The reason why this is so unintuitive is because 0 is the default case in every other part of the system. If something doesn't match a predicate, the value becomes 0. If a value at an index is not defined, it is implicitly zero. A non-matching predicate is 0. This is exploited for reducing values using summation. So the behavior of the system with regards to 0 is always on the mind of the developer. If we add it in another spot, they would think nothing of it. It would be nice if it acted as an identity in a monoidic operation, e.g. as 0 for sums but as 1 for products, but that's not how the system works at all today. And indeed such a thing could be introduced using a special template in place of `c:value-of` that copies the predicates of the referenced value and does the right thing. The _danger_, of course, is that this is _not_ how the system as worked, and so changing the behavior has the risk of breaking something that has relied on undefined behavior for so long. This is indeed a risk, but I have taken some confident in (a) all the test cases for our system pass despite a significant number of x/0=0 being triggered due to limited inputs, and (b) these situations are _not correct today_, resulting in `null` in serialized result data because `JSON.stringify([NaN, Infinity]) === "[null, null]"`. Given all of that, predictable incorrect behavior is better than undefined behavior. So x/0=0 Isn't Bad? ------------------- No, and it's mathematically sound. This decision isn't unprecedented— Coq, Lean, Agda, and other theorem provers define x/0=0. APL originally defined x/0=1, but later switched to 0. Other languages do their own thing depending on what is right for their particular situation. Division is normally derived from a × a⁻¹ = 1, a ≠ 0. We're simply not using that definition—when we say "quotient", or use the `/` symbol, we mean a _different_ function (`div`, in the compiled JS), where we have an _additional_ axiom that a / 0 = 0. And, similarly, 0⁻¹ = 0. So we've taken a _normally undefined_ case and given it a definition. No inconsistency arises. In fact, this makes _sense_ to do, because _this is what we want_. The alternative, as mentioned above, is a lot of boilerplate—checking for 0 any time we want to do division. Complicating the compiler to check for those cases. And so on. It's easier to simple state that, in TAME, quotients have this extra convenient feature whereby you don't have to worry about your denominator being zero because it'll act as though you enclosed it in a case statement, and because of that, all your code continues to operate in an intuitive way. I really recommend reading this blog post regarding the Lean theorem prover: https://xenaproject.wordpress.com/2020/07/05/division-by-zero-in-type-theory-a-faq/
2022-02-24 11:25:15 -05:00
This version includes a backwards-incomplatible change to enable the new
classification system for all packages, which was previously gated behind a
feature flag and the `_use-new-classification-system_` template. (This
template will remain for some time before being removed, but will emit
deprecation warnings.) This new system can be disabled for now by setting
`legacy-classify=true` using the new `TAME_PARAMS` configuration option or
Makefile variable.
Nightly Rust >= 1.53 is now required for TAMER.
Compiler
--------
x/0=0 with global flag for new classification system This was originally my plan with the new classification system, but it was undone because I had hoped to punt on the somewhat controversial issue. Unfortunately, I see no other way. Here I attempt to summarize the reasons why, many of which are specific to the design decisions of TAME. Keep in mind that TAME is a domain-specific language (DSL) for writing insurance rating systems. It should act intuitively for our use case, while still being mathematically sound. If you still aren't convinced, please see the link at the bottom. Target Language Semantics (ECMAScript) -------------------------------------- First: let's establish what happens today. TAME compiles into ECMAScript, which uses IEEE 754-2008 floating-point arithmetic. Here we have: x/0 = Infinity, x > 0; x/0 = -Infinity, x < 0; 0/0 = NaN, x = 0. This is immediately problematic: TAME's calculations must produce concrete real numbers, always. NaN is not valid in its domain, and Infinity is of no practical use in our computational model (TAME is build for insurance rating systems, and one will never have infinite premium). Put plainly: the behavior is undefined in TAME when any of these values are yielded by an expression. Furthermore, we have _three different possible situations_ depending on whether the numerator is positive, negative, or zero. This makes it more difficult to reason about the behavior of the system, for values we do not want in the first place. We then have these issues in ECMAScript: Infinity * 0 = NaN. -Infinity * 0 = NaN. NaN * 0 = NaN. These are of particular concern because of how predicates work in TAME, which will be discussed further below. But it is also problematic because of how it propagates: once you have NaN, you'll always have NaN, unless you break out of the situation with some control structure that avoids using it in an expression at all. Let's now consider predicates: NaN > 0 = false. NaN < 0 = false. NaN === 0 = false. NaN === NaN = false. These will be discussed in terms of classification predicates (matches). We also have issues of serialization: JSON.stringify(Infinity) = "null". JSON.stringify(NaN) = "null". These means that these values are difficult to transfer between systems, even if we wanted them. TAME's Predicates ----------------- TAME has a classification system based on first-order logic, where ⊥ is represented by 0 and ⊤ is represented by 1. These classifications are used as predicates to calculations via the @class attribute of a rate block. For example: <rate-each class="property" generates="propValue" index="k"> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </rate> As can be observed via the Summary Page, this calculation compiles into the following mathematical expression: ∑ₖ(pₖ(tₖ/dₖ)), that is—the quotient is then multiplied by the value of the `property` classification, which is a 0 or 1 respectively for that index. Let's say that tivPropDivisor were defined in this way: <rate-each class="property" generates="tivPropDivisor" index="k"> <!--- ... logic here ... --> </rate> It does not matter what the logic here is. Observe that the predicate here is `property` as well, which means that, if this risk is not a property risk, then `tivPropDivisor` will be `0`. Looking back at `propValue`, let's say that we do have a property risk, and that `buildingTiv` is `[100_000, 200_000]` and `tivPropDivisor` is 1000. We then have: 1(100,000 / 1000) + 1(200,000 / 1000)) = 300. Consider instead what happens if `property` is 0. Since we have no property locations, we have `[0, 0]` as `buildingTiv` and `tivPropDivisor` is 0. 0(0/0) + 0(0/0)) = 0(NaN + NaN) = NaN. This is clearly not what was intended. The predicate is expected to be _strongly_ zero, as if using an Iverson bracket: ((0/0)[0] + (0/0)[0]) = 0. Of course, one option is to redefine TAME such that we use Iverson's convention in place of summation, however this is neither necessary nor desirable given that (a) NaN is not valid within the domain of any TAME expression, and (b) Summation is elegantly generalized and efficiently computed using vector arithmetic and SIMD functions. That is: there's no use in messing with TAME's computational model for a valid that should be impossible to represent. Short-Circuiting Computation ---------------------------- There's another way to look at it, though: that we intended to skip the computation entirely, and so it doesn't matter what the quotient is. If the compiler were smart enough (and maybe one day it will be), it would know that the predicate of `tivPropDivisor` and `propValue` are the same and so there is no circumstance under which we would compute `propValue` and have `tivPropDivisor` be 0. The problem is: that short-circuiting is employed as an _optimization_, and is an implementation detail. Mathematically, the expression is unchanged, and is still invalid within TAME's domain. It is unrepresentable, and so this is not an out. But let's pretend that it was defined that way, which would yield this: { ∑ₖ(pₖ(tₖ/dₖ)), ∀x∈p(x = 1); propValue = < { 0, otherwise. This is the optimization that is employed, but it's still not mathematically correct! What happens if p₀ = 1, but p₁ = 0? Then we have: 1(100,000/1000) + 0(0/0) = 100 + NaN = NaN, but the _intent_ was clearly to have 100 + 0 = 100, and so we return to the original problem once again. Classification Predicates and Intent ------------------------------------ Classifications are used as predicates for equations, but classifications _themselves_ have predicates in the form of _matches_. Consider, for example, a classification that may be used in an assertion to prevent negative premium from being generated: <t:assert failure="premBuilding must not be negative for any index"> <t:match-gte value="premBuilding" value="#0" /> </t:assert> Simple enough—the system will fail if the premium for a given building is below $0. But what happens if premBuilding is calculated as so? <rate-each class="property" yields="premBuildingTotal" generates="premBuilding" index="k"> <c:product> <c:value-of name="propValue" index="k" /> <c:value-of name="propRate" index="k" /> </c:product> </rate-each> Alas, if `property` is false for any index, then we know that `propValue` is NaN, and NaN * x = NaN, and so `premBuilding` is NaN. The above assertion will compile the match into the first-order sentence ∀x∈b(x > 0). Unfortunately, NaN is not greater than, less than, equal to, or any other sort of thing to 0, and so _this assertion will trigger_. This causes practical problems with the `_premium_` template, which has an `@allow-zero@` argument to permit zero premium. Consider this real-world case that I found (variables renamed), to avoid a strawman: <t:premium class="loc" round="cent" yields="locInitialTotal" generates="locInitial" index="k" allow-zero="true" desc="..."> <c:value-of name="premAdditional" /> <c:quotient> <c:value-of name="premLoc" index="k" /> <c:value-of name="premTotal" /> </c:quotient> </t:premium> This appears to be responsible for splitting up `premAdditional` relative to the total premium contribution of each location. It explicitly states that it wants to permit a zero value. The intent of this block is clear: a value of 0 is explicitly permitted and _expected_. But if `premTotal` is for whatever reason 0—whether it be due to a test case or some unexpected input—then it'll yield a NaN and make the entire expression NaN. Or if `premAdditional` or `premLoc` are tainted by a NaN, the same result will occur. The assertion will trigger. And, indeed, this is what I'm seeing with test cases against the new classification system. What about Infinity? Is it intuitive that, should `propValue` in the previous example be positive and `propRate` be 0, that we would, rather than producing a very small value, produce an infinitely large one? Does that match intuition? Remember, this system is a domain-specific language for _our_ purposes—it is not intended to be used to model infinities. For example, say we had this submission because the premium exceeds our authority to write with some carrier: <t:submit reason="Premium exceeds authority"> <t:match-gt name="premBuilding" value="#100k" /> </t:submit> If we had (100,000 / 0) = ∞, then this submit reason would trigger. Surely that was not intended, since we have `property` as a predicate and `propRate` with the same predicate, implying that the answer we _actually_ want is 0! In that case, what we _probably_ want to trigger is something like <rate yields="premFinal"> <t:maxreduce> <c:value-of name="premBuildingTotal" /> <c:value-of name="#500" /> </t:maxreduce> </rate>, in order to apply a minimum premium of $500. But if `premBuildingTotal` is Infinity, then you won't get that—you'll get Infinity, which is of course nonsense. And nevermind -Infinity. Why Wasn't This a Problem Before? --------------------------------- So why bring this up now? Why have we survived a decade without this? We haven't, really—these bugs have been hidden. But the old classification system covered them up; predicates would implicitly treat missing values as 0 by enclosing them in `(x||0)` in the compiled code. Observe this ECMAScript code: NaN || 0 = 0. Consequently, the old classification system absorbed bad values and treated them implicitly as 0. But that was a bug, and had to be removed; it meant that missing indexes in classifications would trigger predicates that were not intended to be triggered, if they matched against 0, or matched against a value less than some number larger than zero. (See `core/test/core/class` for examples.) The new classification system does not perform such defaulting. _But it also does not expect to receive values outside of its valid domain._ Consequently, _NaN and Infinity lead to undefined behavior_, and the current implementation causes the predicate to match (NaN < 0) and therefore fail. The reason for this is because that this implementation is intended to convey precisely the computation necessary for the classification system, as formally defined, so that it can be later optimized even further. Checking for values outside the domain not only should not be necessary, but it would prevent such future optimizations. Furthermore, parameters used to compile into (param||0), to account for missing values or empty strings. This changed somewhat recently with 5a816a4701211adf84d3f5e09b74c67076c47675, which pre-cast all inputs and allowed relaxing many of those casts since they were both wasteful and no longer necessary. Given that, for all practical purposes, 0/0=0 in the system <1yr ago. Infinity, of course, is a different story, since (Infinity||0)=Infinity; this one has always been a problem. Let's Just Fail --------------- Okay, so we cannot have a valid expression, so let's just fail. We could mean that in two different ways: 1. Fail at runtime if we divide by 0; or 2. Fail at compile-time if we _could_ divide by 0. Both of these have their own challenges. Let's dismiss #2 right off the bat for now, because until we have TAMER, that's not really feasible. We need something today. We will discuss that in the future. For #1—we cannot just throw an error and halt computation, because if the `canterm` flag passed into the system is `false`, then _computation must proceed and return all results_. Terminating classifications are checked after returning rather than throwing errors. Since we have to proceed with computation, then the computations have to be valid, and so we're left with the same problem again—we cannot have undefined behavior. One could argue that, okay, we have undefined behavior, but we're going to fail because of the assertion anyway! That's potentially defensible, but it is at the moment undesirable, because we get so many failures. And, relative to the section below, it's not clear to me what benefit we get from that behavior other than making things more difficult for ourselves. Furthermore, such an assertion would have to be defined for every calculation that performs a quotient, and would have to set some intermediate flag in the calculation which would then have to be checked for after-the-fact. This muddies the generated calculation, which causes problems for optimizations, because it requires peering into state of the calculation that may be hidden or optimized away. If we decide that calculations must be valid because we cannot fail, and we have to stick with the domain of calculations, then `x/0` must be _something_ within that domain. x/0=0 Makes Sense With the Current System ----------------------------------------- Let's take a step back. Consider a developer who is unaware that NaN/Infinity are permitted in the system—they just know that division by zero is a bad thing to do because that's what they learned, and they want to avoid it in their code. Consider that they started with this: <rate-each class="property" generates="propValue" index="k"> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </rate> They have inspected the output of `tivPropDivisor` and see that it is sometimes 0. They understand that `property` is a predicate for the calculation, and so reasonably think that they could do something like this: <classify as="nonzero-tiv-prop-divisor" ...> <t:match-ne on="tivPropDivisor" value="#0" /> </classify> and then change the rate-each to <rate-each class="property nonzero-tiv-prop-divisor" ...>. Except that, of course, we know that will have no effect, because a NaN is a NaN. This is not intuitive. So they'd have to do this: <rate-each class="property" generates="propValue" index="k"> <c:cases> <c:case> <t:when-ne name="tivPropDivisor" value="#0" /> <c:quotient> <c:value-of name="buildingTiv" index="k" /> <c:value-of name="tivPropDivisor" index="k" /> </c:quotient> </c:case> <c:otherwise> <c:value-of name="#0" /> </c:otherwise> </c:cases> </rate>. But for what purpose? What have we gained over simply having x/0=0, which does this for you? The reason why this is so unintuitive is because 0 is the default case in every other part of the system. If something doesn't match a predicate, the value becomes 0. If a value at an index is not defined, it is implicitly zero. A non-matching predicate is 0. This is exploited for reducing values using summation. So the behavior of the system with regards to 0 is always on the mind of the developer. If we add it in another spot, they would think nothing of it. It would be nice if it acted as an identity in a monoidic operation, e.g. as 0 for sums but as 1 for products, but that's not how the system works at all today. And indeed such a thing could be introduced using a special template in place of `c:value-of` that copies the predicates of the referenced value and does the right thing. The _danger_, of course, is that this is _not_ how the system as worked, and so changing the behavior has the risk of breaking something that has relied on undefined behavior for so long. This is indeed a risk, but I have taken some confident in (a) all the test cases for our system pass despite a significant number of x/0=0 being triggered due to limited inputs, and (b) these situations are _not correct today_, resulting in `null` in serialized result data because `JSON.stringify([NaN, Infinity]) === "[null, null]"`. Given all of that, predictable incorrect behavior is better than undefined behavior. So x/0=0 Isn't Bad? ------------------- No, and it's mathematically sound. This decision isn't unprecedented— Coq, Lean, Agda, and other theorem provers define x/0=0. APL originally defined x/0=1, but later switched to 0. Other languages do their own thing depending on what is right for their particular situation. Division is normally derived from a × a⁻¹ = 1, a ≠ 0. We're simply not using that definition—when we say "quotient", or use the `/` symbol, we mean a _different_ function (`div`, in the compiled JS), where we have an _additional_ axiom that a / 0 = 0. And, similarly, 0⁻¹ = 0. So we've taken a _normally undefined_ case and given it a definition. No inconsistency arises. In fact, this makes _sense_ to do, because _this is what we want_. The alternative, as mentioned above, is a lot of boilerplate—checking for 0 any time we want to do division. Complicating the compiler to check for those cases. And so on. It's easier to simple state that, in TAME, quotients have this extra convenient feature whereby you don't have to worry about your denominator being zero because it'll act as though you enclosed it in a case statement, and because of that, all your code continues to operate in an intuitive way. I really recommend reading this blog post regarding the Lean theorem prover: https://xenaproject.wordpress.com/2020/07/05/division-by-zero-in-type-theory-a-faq/
2022-02-24 11:25:15 -05:00
- The new classification system is now enabled by default on all packages.
- Legacy systems may use `TAME_PARAMS` (see below) to append
`legacy-classify=true` to disable the new system, for now.
- The new system will consider `x/0=0`; see commit message for detailed
rationale on this change. This will also remove all `Infinity` and `NaN`
values from intermediate and return variables. These get serialized to
`null`s in JSON.
- `TAME_PARMS`, now accepted by the `Makefile` and `configure` script, will
append `key=value` options to the XSLT-based compiler invocations.
- Input mappings will no longer emit the destination param as a dependency.
- `tamed --report` and `TAMED_TUI` for analyzing build performance.
- Runners now store start time and duration for each command, available in
the runpath for reporting.
- `TAMED_RUNTAB_OUT`, if set, will aggregate all runners' runtabs into a
single file as jobs are completed. See `tamed --help` for more
information and examples.
- Improved symbol table processing performance.
- For packages/maps with thousands of dependenices, this may improve
processing time by a minute or more.
Documentation
-------------
- `@mdash` macro now formally defines an argument, correcting errors in
newer versions of Texinfo (~v6.7).
Linker
------
- Remove exception for input map dependency processing (now that compiler no
longer emits such a dependency).
- Reduce peak memory usage by clearing buffer of `xmlo` reader between
events.
- How effective this is varies depending on the size of individual
entities within the XML document. In some cases, it can reduce peak
memory usage by half.
Tools
=====
- `build-aux/check-coupling` will now prevent `supplier/` packages from
importing `ui/` packages; previously only the reverse was true.
2021-07-21 15:05:52 -04:00
v18.0.3 (2021-07-21)
====================
2021-07-21 15:04:59 -04:00
This release significantly improves the performance of executables
containing large constants, and fixes an optimization-related bug introduced
in v18.0.0.
Compiler
--------
- Place constants into static section in linked executable.
- This was the case in the old linker before the `tameld`
proof-of-concept. The benefits are significant when large constants are
used (e.g. for large tables of data).
- Do not report value list optimization error on duplicate conjunctive
predicates.
- This doesn't emit code any differently, it merely permits the situation,
which can occur in generated code.
2021-07-15 23:50:53 -04:00
v18.0.2 (2021-07-15)
====================
2021-07-15 23:50:00 -04:00
This is a bugfix release that corrects issues with the Summary Page compiler
and corrects behavior with the new classification system (that is currently
off unless explicitly requested).
Compiler
--------
- Make Summary Page less chatty.
- Fix incorrect package name for generated worksheet packages.
- Restrict `TRUE`-match optimization to classification matches (class
composition).
- This was mistakenly not considering the domain of the match, and
therefore was applying the optimization in situations where it should
not. Results of previous classifications are currently the only place
we guarantee a boolean value.
- Apply classification alias optimization to any `1`-valued constant match.
- Previously applied only to `TRUE`.
Summary Page
------------
- Correctly generate input fields for params using imported types.
- This is a long-standing (nearly 10-year-old) bug.
2021-06-24 10:37:25 -04:00
v18.0.1 (2021-06-24)
====================
This is a minor maintenance release.
Compiler
--------
- Remove internal notice when new system is used to emit code for a
particular classification.
2021-06-23 12:54:25 -04:00
v18.0.0 (2021-06-23)
====================
This release focuses primarily on compiler optimizations that affect runtime
performance (both CPU and memory). The classification system has undergone
a rewrite, but the new system is gated behind a template-based feature flag
`_use-new-classification-system_` (see Core below). Many optimizations
listed below are _not_ affected by this toggle.
Compiler
--------
- Numerous compiler optimizations including (but not limited to):
- Classification system rewrite with significant correctness and
performance improvements, with significantly less generated code.
- There is more work to be done in TAMER.
- This change is gated behind a feature toggle (see
`_use-new-classification-system_` in Core below).
- Significant reduction in byte count of JavaScript target output.
- Classifications with single-`TRUE` predicate matches are aliases and now
compile more efficiently.
- Classifications that are a disjunction of conjunctions with a common
predicate will have the common predicate hoisted out, resulting in more
efficeint code generation.
- Classifications with equality matches entirely on a single param are
compiled into a `Set` lookup.
- Most self-executing functions in target JavaScript code have been
eliminated, resulting in a performance improvement.
- Floating point truncation now takes place without using `toFixed` in
JavaScript target, eliminating expensive number->string->number
conversions.
- Code paths are entirely skipped if a calculation predicate indicates
that it should not be executed, rather than simply multiplying by 0
after performing potentially expensive calculations.
- A bunch of wasteful casting has been eliminated, supplanted by proper
casting of param inputs.
- Unnecessary debug output removed, significantly improving performance in
certain cases.
- Single-predicate any/all blocks stripped rather than being extracted
into separate classifications.
- Extracted any/all classifications are inlined at the reference site when
the new classification system is enabled, reducing the number of
temporaries created at runtime in JavaScript.
- Summary Page now displays values of `lv:match/@on` instead of debug
values.
- This provides more useful information and is not subject to the
confusing reordering behavior of the compiler that is not reflected on
the page.
- Changes that have not yet been merged will remove debug values for the
classification system.
Core
----
- New feature flag template `_use-new-classification-system_`.
- This allows selectively enabling code generation for the new
classification system, which has BC breaks in certain buggy situations.
See `core/test/core/class` package for more information.
- Remove `core/aggregate`.
- This package is not currently utilized and is dangerous---it could
easily aggregate unintended values if used carelessly. Those who know
what they are doing can use `sym-set` if such a thing is a good thing
within the given context, and proper precautions are taken (as many
templates already do today).
Rust
----
- Version bump from 1.42.0 to 1.48.0 now that intra-doc links has been
stabalized.
Miscellaneous
-------------
- `build-aux/progtest-runner` will now deterministically concatenate files
based on name rather than some unspecified order.
2021-05-27 13:22:00 -04:00
v17.9.0 (2021-05-27)
====================
2021-05-10 14:21:24 -04:00
This is a documentation/design release, introducing The TAME Programming
Language in `design/tpl`.
Compiler
-------
- Allow the mapping of flag values from `program.xml`.
2021-05-10 14:21:24 -04:00
Design
------
- Introduce The TAME Programming Language.
2021-03-18 09:56:02 -04:00
v17.8.1 (2021-03-18)
====================
2021-03-18 09:55:36 -04:00
This release contains a bufix for recent build changes in v17.8.0 that were
causing, under some circumstances, builds to fail during dependency
generation. It also contains minor improvements and cleanup.
Build System
------------
2021-03-18 09:55:36 -04:00
- [bugfix] Lookup tables will no longer build `rater/core/vector/table` when
geneating the `xml` package.
- This was causing problems during `suppliers.mk` dependency generation.
The dependency is still in place for the corresponding `xmlo` file.
- This was broken by v17.8.0.
- Minor improvements to `tame` and `tamed` scripts to ensure that certain
unlikely failures are not ignored.
2021-03-18 09:55:36 -04:00
2021-02-23 10:51:59 -05:00
v17.8.0 (2021-02-23)
====================
This release contains changes to the build system to accommodate
liza-proguic's introduction of step-based packages (in place of a monolithic
`package-dfns.xml`), as well as miscellaneous improvements.
Compiler
--------
- `rater.xsd`, used for certain validations of TAME's grammar, has been
updated to an out-of-tree version; it had inadvertently gotten out of
date, and the discrepency won't happen again in the future.
- Further, limits on the length of `@yields` identifiers have been
removed; the lack of namespacing and generation of identifiers from
templates can yield longer identifier names.
Build System
------------
- Only modify `.version.xml` timestamp when hash changes. This allows
its use as a dependency without forcefully rebuilding each and every time.
- `configure` will no longer immediately generate `suppliers.mk`.
- Additionally, `build-aux/suppmk-gen`, which `configure` directly invoked
until now, was removed in favor of generic rules in `Makefile.am`.
- Step-level imports in program definitions are now recognized to
accommodate liza-proguic's step-level package generation.
- Step-level program packages are now properly accounted for as dependencies
for builds.
- `supplier.mk` is now automatically regenerated when source files
change. This previously needed to be done manually when imports changed.
- `supplier.mk` generation will no longer be verbose (it'll instead be
only one line), which makes it more amenable to more frequent
regeneration.
2020-12-09 09:59:09 -05:00
v17.7.0 (2020-12-09)
====================
This release provides tail-call optimizations aimed at the query system in
core.
Compiler
--------
- [bugfix] Recursive calls using TCO will wait to overwrite their function
arguments until all expressions calculating the new argument values have
completed.
`tame-core`
-----------
- `mrange` is now fully tail-recursive and has experimental TCO applied.
- It was previously only recursive for non-matching rows.
2020-12-03 14:09:37 -05:00
v17.6.5 (2020-12-03)
====================
This release improves Summary Page performance when populating the page with
data loaded from an external source.
Summary Page
------------
- Populating the DOM with loaded data now runs in linear time.
2020-11-23 15:26:54 -05:00
v17.6.4 (2020-11-23)
====================
This release tolerates invalid map inputs in certain circumstances.
Compiler
--------
- Tolerate non-string inputs to `uppercase` and `hash` map methods.
2020-11-03 13:24:36 -05:00
v17.6.3 (2020-11-03)
====================
- Update the CDN used to get MathJax.
2020-10-01 10:22:17 -04:00
v17.6.2 (2020-10-01)
====================
- Optionally include a "program.mk" file if it is present in the project's root
directory. This allows us to move program specific tasks outside of TAME.
2020-09-23 16:32:19 -04:00
v17.6.1 (2020-09-23)
====================
Compiler
--------
- `lv:param-class-to-yields` will now trigger a failure rather than relying
on propagating bad values, which may not result in failure if the symbol
is represented by another type (non-class) of object.
Miscellaneous
-------------
- `package-lock.json` additions.
2020-08-19 15:30:00 -04:00
v17.6.0 (2020-08-19)
====================
This release provides a new environment variable for JVM tuning. It does
not provide any new compiler features or performance enhancements in itself,
though it enables optimizations through JVM tuning.
Compiler
--------
- The new environment variable `TAMED_JAVA_OPTS` can now be used to provide
arguments to the JVM. This feature was added to support heap ratio
tuning.
Miscellaneous
-------------
- `build-aux/lsimports` was causing Gawk to complain about the third
argument to `gensub`; fixed.
- `bootstrap` will test explicitly whether `hoxsl` is a symbol link, since
`-e` fails if the symlink is broken.
2020-07-15 11:16:14 -04:00
v17.5.0 (2020-07-15)
====================
This release adds support for experimental human-guided tail call
optimizations (TCO) to resolve issues of stack exhaustion during runtime for
tables with a large number of rows after having applied the first
predicate. This feature should not be used outside of `tame-core`, and will
be done automatically by TAMER in the future.
`tame-core`
-----------
- `vector/filter/mrange`, used by the table lookup system, has had its
mutually recursive function inlined and now uses TCO.
- This was the source of stack exhaustion on tables whose predicates were
unable to filter rows sufficiently.
Compiler
--------
- Experimental guided tail call optimizations (TCO) added to XSLT-based
compiler, allowing a human to manually indicate recursive calls in tail
position.
- This is undocumented and should only be used by `tame-core`. The
experimental warning will be removed in future releases if the behavior
proves to be sound.
- TAMER will add support for proper tail calls that will be detected
automatically.
2020-07-02 12:48:16 -04:00
v17.4.3 (2020-07-02)
====================
2020-07-02 12:47:12 -04:00
This release fixes a bug caused by previous refactoring that caused
unresolved externs to produce an obscure and useless error for the end
2020-07-02 12:47:12 -04:00
user.
Linker
------
- Provide useful error for unresolved identifiers.
- This was previously falling through to an `unreachable!` block,
producing a very opaque and useless internal error message.
2020-05-13 08:09:48 -04:00
v17.4.2 (2020-05-13)
====================
This release adds GraphML output for linked objects to allow us to
inspect the graph.
Linker
------
- Add `--emit` oprion to `tamer/src/bin/tameld.rs` that allows us to specify
the type of output we want.
- Minor refactoring.
Miscellaneous
-------------
- Added `make` target to build linked GraphML files.
- Updated `make *.xmle` target to explicitly state it is emitting `xmle`.
- Added Cypher script to use in Neo4J after a GraphML file is imported.
2020-04-29 15:49:57 -04:00
- `RELEASES.md`
- Add missing link to semver.org.
- Fix `tame-core` heading, which was erroneously Org-mode-styled.
- Rephrase and correct formatting of an introduction paragraph.
2020-04-29 15:48:21 -04:00
2020-04-29 15:34:29 -04:00
v17.4.1 (2020-04-29)
====================
This release refactors the linker, adds additional tests, and improves
errors slightly. There are otherwise no functional changes.
Compiler
--------
- Refactor proof-of-concept dependency graph construction code.
- Improvements to error abstraction which will later aid in reporting.
Miscellaneous
-------------
- `RELEASES.md` added.
- `tools/mkrelease` added to help automate updating `RELEASES.md`.
- `build-aux/release-check` added to check releases.
- This is invoked both by `tools/mkrelease` and by CI via
`.gitlab-ci.yml` on tags.
v17.4.0 (2020-04-17)
====================
This release focuses on moving some code out of the existing XSLT-based
compiler so that the functionality does not need to be re-implemented in
TAMER. There are no user-facing changes aside form the introduction of two
new templates, which are not yet expected to be used directly.
2020-04-29 15:49:57 -04:00
`tame-core`
-----------
- New `rate-each` template to replace XSLT template in compiler.
- New `yields` template to replace XSLT template in compiler.
- Users should continue to use `rate-each` and `yields` as before rather
than invoking the new templates directly.
- The intent is to remove the `t` namespace prefix in the future so that
templates will be applied automatically.
Compiler
--------
- XSLT-based compiler now emits `t:rate-each` in place of the previous XSLT
template.
- XSLT-based compiler now emits `t:yields` in place of the previous XSLT
template.