This tests the existing state of TokenDao before additional modifications
are made. This commit also further refines the types introduced in a
previous commit.
This is also the first test written in Typescript.
* package.json.in (devDependencies): Add node, chai, and mocha types.
* src/server/token/TokenDao.ts (updateToken): `data` accepts null (as it
should). Do not conditionall add data to object (it doesn't matter for
later retrieval). Note nondeterminism with date. More concise syntax
for object fields.
* src/server/token/TokenQueryResult.ts: Make all fields readonly.
(TokenStatus): Date is no longer optional (see above mention).
* src/types/mongodb.d.ts: Remove generics (erroneously added).
(Collection)[update]: Remove 3-argument declaration (see comment).
* test/server/token/TokenDaoTest.ts: New test case.
This was previously hard-coded to "exports", which is so named because the
system is currently used only to export data to another system.
This change retains the previous functionality.
* src/server/daemon/controller.js (_initExportService): Pass new TokenDao
argument.
* src/server/token/TokenDao.ts (TokenDao)[_rootField]: New property.
(constructor, getToken, _genRoot): Use it.
This adds type information for both MongoCollection and some details about
the structure of the data itself as it is stored in the database.
* src/server/token/TokenDao.ts: Import types from TokenQueryResult.
(TokenType): Move to TokenQueryResult module.
(TokenData)[status]: string=>TokenStatus.
(TokenDao)[_collection]: Remove optional type specifier.
[_collection]: Make readonly.
[updateToken]: Use TokenStatus type for token_entry.
[getToken]: Add return type. Add type for query result. Remove useless
length check (no such field exists). Add types for variables.
[_getLatestToken]: Use interface instead of inline type definition.
[_getRequestedToken]: Use type definition for `data' instead of
`any'. Add type of variable.
* src/server/token/TokenQueryResult.ts: Define structure of data in Mongo.
* src/types/mongodb.d.ts: Add type information for methods used by TokenDao.
This is no longer going to be coupled with the concept of a "service",
though it'll continue to be used by it.
* src/server/daemon/controller.js: Change TokenDao import path.
* src/server/service/TokenedService.js: Likewise.
* src/server/service/TokenDao.ts: Rename (delete).
* src/server/token/TokenDao.ts: New name.
This is the first test of integrating typescript. I'm using the `export ='
syntax to keep imports the same using `require', but note that the `new'
keyword is now required for instantiation (whereas it was not with easejs).
The quicksave feature was added back in 2011 when the framework was somewhat
unstable---we were taking calls from users and it wasn't a pleasent
experience to tell them that they had to refresh the page to work around
some issue with bad state, and lose all unsaved data.
This feature is now more trouble than it's worth, since it causes a number
of bugs and can even cause data corruption. If this is reintroduced in the
future, I'd prefer it be done by periodically saving quote state after every
or a few bucket modifications, to permit Meteor-like features.
This keeps the feature available on the client as a heartbeat for quote
locking; this can be removed in the future when we handle distributed
locking.
* src/client/Client.js (_changeQuote): Stop setting quicksave data from
request.
(_mergeQuickSaveData): Remove now-unused method.
(_createUi): Stop merging quicksave data on render.
(saveStaging): Add note that this method is now only used as a heartbeat.
* src/client/quote/ClientQuote.js (saveStaging): Add comment indicating that
this is now used for a heartbeat.
(setQuickSaveData, getQuickSaveData): Remove methods.
* src/quote/BaseQuote.js (_quickSaveData): Remove field.
(setQuickSaveData, getQuickSaveData): Remove methods.
* src/server/Server.js (initQuote): Do not create or initialize quicksave
data.
(handleQuickSave): Remove method.
* src/server/daemon/controller.js (doRoute): Return empty reply without
processing quicksave data. Continue touching session to retain quote
lock.
* src/server/db/MongoServerDao.js (saveQuote): Do not clear quicksave.
(quickSaveQuote): Remove method.
The PHP objects in the session from PHP7 cannot be unserialized. Since
they are not needed, we can safely ignore them.
Co-Authored-By: Jim Grundner <james.grundner@rtspecialty.com>
Prior to this change, a single generic message was always shown simply
stating that the quote had been locked. These changes now allow for
different messages to be displayed in different circumstances.
This now publishes to a fanout exchange instead of a queue, which allows
consumers to handle their own queue configuration.
This also adds some basic logging that was missing from the first version.
Note that I still don't consider this to be production-quality code; it's
missing tests, and there's still notes that need to be addressed.
This is unfortunately not production-ready code, but we need to get
something out there in the meantime. The RatingServicePublish's docblock
mentions some of the shortcomings, which will be addressed in the near
future.
There is also more documentation to come once we settle on an implementation.
DEV-4400
It looks like the metabucket is never initialized, so saving the quote is
right now the only thing that sets default values. That should be fixed in
the future.
This also begins adding tests for the terrible MongoServerDao, that could
use some refactoring.
* src/server/db/MongoServerDao.js: Make `meta' mutable. I had forgotten to
remove the code that mutates it (since our version of v8 right now does
not blow up for const assignments), so this is all that's needed.
* test/server/db/MongoServerDaoTest.js: New file to test this situation.
The metabucket expects vectors, not scalars. This causes the value
to be inaccessible when loaded by the ProcessManager for rating,
because it uses the Bucket implementation expecting vectors.
This wasn't noticed at first because it works through TAME's map.
* doc/bucket.texi (Metabucket): Clarify expected data format.
* src/server/db/MongoServerDao.js (saveQuote): Store
`meta.liza_timestamp_initial_rated' as a vector.
This value already existed on the document, but was inaccessible to external
systems. This is now accessible to e.g. raters.
* doc/bucket.texi (Metabucket): New section.
* src/server/db/MongoServerDao.js (saveQuote): Set initial quoted date as
liza_timestamp_initial_rated. Update metabucket keys individually so as
not to inadvertently overwrite the entire metabucket.
DEV-3715
Default data was converted to an empty array if the data evaluated to
false. We only want to convert it if it is undefined so values that are
false remain false.
The postProcessResults method occasionally failed to be overridden in
local environments due to issues with how npm handled its dependencies.
For some reason, changing this to a public method corrected the issue. I
am changing this to, hopefully, prevent future frustrations.
* src/program/ProgramInit.js (_isKnownType): Account for ancient qtype
representation (as a string).
* src/server/quote/ProgramQuoteCleaner.js (_isKnownType): Likewise.
* test/program/ProgramInitTest.js: New test case for this situation.
* test/server/quote/ProgramQuoteCleanerTest.js: Modify existing test case
for this situation.
These denote fields that are generated but do not actually have any data
associated with them. For example, select options with predicates have a
field generated so that they contribute to the group field count (so that
the group will automatically show/hide appropriately), but those should
never have values associated with them in the bucket.
This was manifesting as a nasty bug: The bucket contained a key for
generated options. When the quote is loaded, the client "empties" the
bucket. In doing so, it set the option value to the empty string, which had
the effect of rendering the dropdown useless---every value was the empty
string!
* src/program/ProgramInit.js (_isKnownType): New method.
(init): Use it and ignore fields with unknown types.
* src/server/Server.js: Add note that we shouldn't have this logic
duplicated between ProgramInit and ProgramQuoteCleaner.
* src/server/quote/ProgramQuoteCleaner.js (_fixGroup): Ignore fields with
unknown types.
(_isKnownType): New method.
* test/program/ProgramInitTest.js: Update existing tests. Add new.
* test/server/quote/ProgramQuoteCleanerTest.js: Test this case.
* src/server/quote/ProgramQuoteCleaner.js (clean): Add docblock.
Replace previous linked group cleaning with call to `_fixGroup'.
(_fixGroup): New method. Similar logic to previous linked group cleaning,
except that fields are never truncated.
(_fixLinkedGroups, _getLinkedIndexLength): Remove methods.
(_getGroupLength): New method determining group size from leader length,
which also accounts for linked groups.
* test/server/quote/ProgramQuoteCleanerTest.js: New test case.
* src/server/daemon/Daemon.js (getProgramController): Do not initialize
`no_results_url' to an empty string if not provided; instead, keep any
existing value. Otherwise, later calls with no arguments will clear
previously-set values.
DEV-3254
If the submission failed, we probably want to try again next time around.
* src/server/service/RatingServiceSubmitNotify.js
(_maybeNotify): Extract logic from `#postProcessRaterData'. Only set
notification flag in absence of dapi error.
(postProcessRaterData): Use it.
* test/server/service/RatingServiceSubmitNotifyTest.js: Update tests
accordingly.
Previously, it was just trying to serialize an object, which was useless.
* src/server/Server.js (_formatValidationFailures): New method.
(_doQuoteSave): Use it.
Requests that came in between (for example) a post request and the actual
validation of the posted data would potentially reset the flag, causing
internal assertions to fail.
* src/server/Server.js
(sendStep): Provide session to #_forwardValidate.
(_forwardValidate): Set `program.isInternal' immediately before invoking
event. Provide session to #quoteFill.
[session]: New argument.
(quoteFill): Add `session' argument. Provide session to hooks.
(loadProgram): Receive session via #quoteFill hook. Set
`program.isInternal' before submit hook.
* src/server/daemon/controller.js (doRoute): Remove `program.isInternal'
set.
This is not the ideal implementation. Instantiation of RatingService and
such should be moved out of the controller entirely, but we don't have the
time for that right now.
* src/server/db/MongoServerDao.js
(getDocumentField,setDocumentField): New methods.
* src/server/service/RatingServiceSubmitNotify.js
(postProcessRaterData): Only notify when notification flag is not set.
(_getNotifyState, _setNotified): New methods.
* test/server/service/RatingServiceSubmitNotifyTest.js: Modify accordingly.
This wasn't intended to make its way into a public repo. :)
The existing key was a long-forgotten kluge that was supposed to be
temporary, allowing internal services to create quotes without
authentication. The chances of this being practically exploited are minimal
in our environment, and it's auditable using webserver logs.
This moves the skey into a configuration file, which allows it to vary by
server and be rotated until a better solution is made available. skey is
disabled by default (empty string), and when used by us internally, the keys
are now generated using a CSPRNG rather than a brute-forcable 5-byte key
that was hard-coded.
The fact that this appears in webserver logs is a big issue as well. I
added a task to address that.
* conf/vanilla-server.json (skey): New key. Default empty.
* src/server/daemon/Daemon.js (start): Provide skey to `#getRouters'.
(getRouters): Provide skey to `#getProgramController'.
(getProgramController): Set skey on `controller'.
* src/server/daemon/controller.js (skey): New mutable export (unideal; quick
change).
(has_skey): Use it.
quicksave data wouldn't always be cleared on save, meaning it might not be
cleared until the next _empty_ quicksave from the client.
* src/server/db/MongoServerDao.js (saveQuote): Always clear quicksave.
Was outputting NaN, because the property that was being used no longer
existed after previous changes.
* src/server/daemon/Daemon.js (_initHttpServer): Use HTTP port number
in output.
Previous to this change, we'd write the clear to the database, but
then immediately after, the quote save would restore it. Oops.
* src/server/Server.js (_monitorMetadataPromise): Clear data in
metabucket rather than directly invoking a DB call. This will be
applied when the quote is saved.
This mixes in support for non-terminating nulls. It would have been
nice to handle that in a separate commit for clarity, but the
refactoring came as a consequence of trying to provide a working
implementation.
Various inconsistencies and subtle bugs in unlikely situations have
been fixed by this, including modifying objects passed as arguments to
various methods, and inconsistent handling of diff data.
Changes are more consistently recognized. Perhaps the most noticeable
consequence is that moving between steps no longer prompts to discard
changes---previously calculated values would trigger the dirty flag on
steps even if the user didn't actually change anything. I (and
others) have wanted this fixed for many years.
This is a very dense commit that touches a core part of the
system. Hopefully the Changelog below helps.
* src/bucket/Bucket.js
(setValues): [BC-BREAK] Remove parameters `merge_index' and
`merge_null' parameters.
* src/bucket/DelayedStagingBucket.js
(setValues): [BC-BREAK] Remove `merge_index' and `merge_null
parameters. Remove distinction between `merge_index' and non-.
* src/bucket/QuoteDataBucket.js
(setValues): [BC-BREAK] Remove `merge_index' and `merge_null
parameters. Remove respective arguments from `_mergeData' call.
(_mergeData): Remove same parameters. Remove handling of
`merge_index' and `merge_null'.
(overwriteValues): Append `null' to each vector.
* src/bucket/StagingBucket.js
(_initState): Use `Object.create' instead of explicit prototype
instantiation (functionally equivalent).
(merge): Minor comment correction.
(_hasChanged): Rename to `_parseChanges'.
(_parseChanges): Rename from `_hasChanged'. Remove `merge_index'
parameter. Generate new object rather than mutation original
data (prevent dangerous and subtle bugs from side-effects). Clone
each vector rather than modifying/referencing directly (this was
previously done during merge). Remove `merge_index'
distinction. Handle non-terminating `null' values.
(setValues): [BC-BREAK] Remove `merge_index' and `merge_null'
parameters. Use new object generated by `_parseChanges'. Remove
cloning of each vector (`_parseChanges' now does that). Remove
`merge_index' distinction.
(overwriteValues): Remove argument to `setValues' call.
(getFilledDiff): [BC-BREAK] Use `_staged' rather than `_curdata'.
(commit): Remove second and third arguments of call to `setValues'
of underlying bucket.
* src/client/Client.js
(_initStepUi): Remove second argument of calls to quote `setData'.
* src/client/quote/ClientQuote.js
(setData): [BC-BREAK] Remove `merge_nulls' parameter. Remove second
and third arguments of call to staging bucket `setValues'. Add
comment indicating a long-standing problem with committing the
staging bucket contents before save has succeeded.
* src/server/request/DataProcessor.js
(processDiff): Remove `permit_null' argument of `sanitizeDiff'
call.
(sanitizeDiff): Remove `permit_null' parameter. Hard-code filter
call's `permit_null' argument to `true'.
(_determineDapiFields): Properly handle `null's (ignore) rather than
inadvertently converting them into the string "null".
* test/bucket/StagingBucketTest.js: Modify test cases
accordingly. Add tests to verify that updates and diffs operate
as expected, especially support for non-terminating `null's.
(createStubBucket): Use `QuoteDataBucket'. Ideally remove this
coupling in the future, but this is a more realistic test case for
the time being.
* test/server/request/DataProcessorTest.js: Update test to account for
hard-coded `given_null' argument.
I'm not sure if this has ever caused real problems, but this was
dangerous.
* src/server/service/RatingService.js (_performRating): Store cleaned
rating data in separate variable to ensure cleaned data are not
saved in place of the actual complete response.
This represents a portion of the refactoring that I had intended to do
until I realized that there was a simpler solution to the problem that
we were having (having proguic add stored calculated values to the
defaults object).
So ideally we'll continue extracting all quote init code out of
`Server' and into `ProgramInit' in the future.
* doc/server.texi (Liza Server): Mention `ProgramInit'.
* src/program/ProgramInit.js: Add class.
* src/server/DocumentServer.js: Use it.
* src/server/Server.js (_progInit): Add private field.
(__construct): Accept ProgramInit instance and assign to field.
(initQuote): Use promise returned by `_getDefaultBucket'.
(_getDefaultBucket): Proxy to `ProgramInit#init', which returns a
promise.
This was a bit involved because the system had to be made async all
the way up the stack. No attempt was made to clean up the mess up the
stack---no time.
* src/dapi/DataApiFactory.js
(fromType): [BC BREAK] Fix docblock. Add `api_name' param. Call
`#descLookup' and return promise.
(descLookup): Add method. Return promise resolving to provided
descriptor. Intended to be overridden by subtype.
* src/dapi/DataApiManager.js
(_dataApis): Update docblock to indicate that it now stores
promises.
(getApiData): Expect promise for `DataApiFactory#fromType' call.
* src/server/DocumentServer.js: (create): [BC BREAK] Accept
configuration. Look up dapi conf and pass to
`ServerDataApiFactory' ctor. Return promise.
* src/server/daemon/Daemon.js (_initRouters): Provide configuration.
* src/server/daemon/controller.js
(init): Accept configuration. Handle return of promise from
`_createDocumentServer'.
(_createDocumentServer): Accept configuration, providing to
`DocumentServer#create'. Because of aforementioned change to
`#create', returns promise.
* src/server/request/ServerDataApiFactory.js: Add StoreMissError
import.
(_conf): Add property.
(constructor): [BC BREAK] Accept configuration.
(descLookup): Add override. Look up configuration for provided
dapi.
This is based (very) loosely on an internal script to start the
daemon. It accepts a configuration and starts the daemon.
To accommodate the configuration, a number of miscellaneous changes
have been made.
The vanilla configuration shows the concept, but it has not yet been
fully implemented; that'll likely happen at a later date. Until then,
the existing environment-variable-based configuration will be used.
* bin/server.js: Add file.
* conf/vanilla-server.json: Example configuration added.
* src/server/daemon/Daemon.js (_httpPort): Remove field.
(_conf): Add field.
(__construct): [BC BREAK] Accept conf instead of port and log
priority. Move initialization code into `start'.
(start): [BC BREAK] Initialization code moved here. Now returns
promise for entire daemon, which will error in the event of an
error starting. Move existing code into `_startDaemon'.
(_startDaemon): Old `start' code. Invoked after `start'
initialization.
(_createDebugLog, _createAccessLog): Use configuration. Return
promise.
(_initHttpServer): Use configuration.
(_httpError): Add function to output error and exit. Extracted from
`_initHttpServer'.
* src/server/daemon/scripts.js: [BC BREAK] Append "program/" to
`LV_LEGACY_PATH' so that it can be re-used for script lookups
rather than using the cwd. This removes the need of the cwd being
the legacy src path.
I wish I knew this option existed ~7yr ago. This should have been
done long ago, but we've had so few problems with it, that it had
never been explored.
This [version of this] mongo library is ancient and needs to be
upgraded. Until then, this'll do.
* src/server/daemon/controller.js (init): Set `auto_reconnect'.
`undefined' was inserted into the session if data could not be
retrieved.
* src/server/request/UserSession.js (_appendSessionData): Do not write
session data if original data cannot be retrieved.
This is a terrible kluge, but time doesn't permit modifying the
system. All of this also touches old code that is untested, which is
difficult to modify with confidence.
* src/server/DocumentServer.js (DocumentServer#create): Use
StagingBucket.
* src/server/Server.js: Remove logic now handled by DataProcessor.
* src/server/request/DataProcessor.js (processDiff): Wrap in
StagingBucket to filter out values that do not result in changes.
* test/server/request/DataProcessorTest.js: Update failing cases.
Prefixes with "meta:".
* src/server/Server.js (initQuote): Initialze metadata to empty.
* src/server/daemon/controller.js (createQuote): Add metabucket.
* src/server/db/MongoServerDao.js (saveQuote): Add additional
documentation to docblock. Save all metadata if no bucket data
provided (which indicates a full quote save). This happens on
version change.
* src/server/rater/ProcessManager.js (_genData): Add method to merge
metadata into bucket data with "meta:" prefix.
(byId): Use merged data.
* src/server/daemon/Daemon.js (__construct): Reference new class.
* src/server/rater/ProcessManager.js: Add class (extracted from
service).
* src/server/rater/service.js: Refactored into ProcessManager.
It never was a thread; it's always been a process.
* src/server/rater/thread.js: Renamed.
* src/server/rater/process.js: Renamed from thread.js.
* src/server/rater/service.js: s/thread/process/.
This reverts CapturedUserResponse to the state it was in prior to
being extracted into liza. The issue was supposed to have been
resolved by now, but I haven't wanted to make an ease.js release until
the company that bought the company I work for signs the copyright
waiver. Which has been slow-going, to say the least.
The problem is that it is not recognizing a named trait extending a
class as a parameterized trait (it's not seeing __mixin).
* src/server/request/CapturedUserResponse.js: Use interface.
* src/server/request/IProtUserResponse.js: Add temporary interface.
What a cluster.
This was a lot of work to work around existing, bad APIs; there is no
time to refactor at the moment; this already took much longer than
expected.
Copyright notices updated. More casual references to "LoVullo
Associates" replaced with "RT Specialty / Lovullo", which will be "RT
Specialty Buffalo" in the future. Or "RT Specialty", depending on how
this is rolled out. Or "Ryan Specialty Group". Who knows.
"R-T Specialty, LLC." is the legal name, which includes the dash. Not
to be confused with a certain television network.
I have sat on releasing a lot of this code for years because I wanted
the liza repo to be in a pristine state---tests and all---which
required a great deal of refactoring. Well, that never happened, and
time is up.
LoVullo Associates---my employer---has been purchased by another
company. This means that any agreement with LoVullo regarding
releasing free software is going to have to be re-negotiated with this
new company, and I have no idea how those negotiations will go. So,
I have no choice but to simply release everything in its current state,
or risk it being lost forever.
This represents work over the past 6--7 years, 99.9% of it written by
me. This project has been my baby for quite some time, and has been
through a number of battles with deadlines and other unfortunate
circumstances; the scars show. I also didn't really "know" JS when
starting this project. Perhaps you can help improve upon it.
There are some odds-and-ends that could be committed. And references
to insurance and LoVullo need to be removed to generalize this.
I hope that this will not be the last public commit for this project.
I'll fight the good fight and we'll see where that takes us. Maybe
it'll be easy.
Happy hacking.
Mike Gerwitz
Mike Gerwitz