1
0
Fork 0
Commit Graph

231 Commits (3fbe9b31ff9b0e4700332fa23168ecc7ad0a09cc)

Author SHA1 Message Date
Austin Schaffer 3fbe9b31ff [DEV-6763] Include ratedata in data deltas 2019-12-26 15:28:46 -05:00
Joseph Frazer aa2ba885af [DEV-6756] Prevent special chars from "local-part" of email address
The `"(),:;<>@[\]`` are allowed in the "local-part" of an email address
under certain circumstances, but our regex is not checking those
circumstances. It is better if we do not allow them until we improve
validation.
2019-12-23 08:26:54 -05:00
Austin Schaffer 9f22db72fd [DEV-5312] Use milliseconds for timestamp 2019-12-19 14:42:10 -05:00
Austin Schaffer e2a9d17dc6 [DEV-5312] Do not use Object.create when applying delta 2019-12-19 14:42:10 -05:00
Austin Schaffer 1cb47d4acf [DEV-6710] [WIP] Save environment on quote save 2019-12-19 14:17:06 -05:00
Austin Schaffer 5e5932fe9d [DEV-5312] Get delta processor program field from mongodb 2019-12-18 11:36:40 -05:00
Austin Schaffer d7f01e12c8 [DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores 2019-12-18 09:48:26 -05:00
Austin Schaffer 699ee29a95 [DEV-5312] Use formatted time when logging events 2019-12-17 11:59:06 -05:00
Austin Schaffer 542745da8a [DEV-5312] Log quote_id along with doc_id 2019-12-17 11:59:06 -05:00
Austin Schaffer ccfd40c64f [DEV-5312] Allow for arbitrary depth in bucket schema 2019-12-17 11:59:01 -05:00
Austin Schaffer b6f72b4b09 [DEV-5312] Create additional indexes and process less frequently 2019-12-16 09:27:29 -05:00
Austin Schaffer e3dded760d [DEV-5312] Define document meta data and restructure message writer.
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9eb1f3afca DeltaProcessor: Correct delta application
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 819701eca3 DeltaProcessor: Encapsulate getDeltas and remove redundant tests
getDeltas should be encapsulated.  It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz d0a3ba573f DeltaProcessor: Remove redundant tests
These are covered by other tests and are testing too closely to the
implementation.  This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 84c2601f1c EventMediator: Log error stack under context 2019-12-12 10:27:09 -05:00
Mike Gerwitz e885026e0a DeltaPublisher: Remove parsing from constructor 2019-12-12 10:27:09 -05:00
Mike Gerwitz 4383d15c5a AvroFactory: Extract avro-js type definitions into src/types
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9d6cb23e16 AmqpConnection: Propagate assertExchange promise
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer d9ee999adb [DEV-5312] Refactor class dependencies
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer 1ddb6f29eb [DEV-5312] Add logic to apply delta to bucket then publish modified bucket 2019-12-12 10:13:27 -05:00
Austin Schaffer 5ee9a5d340 [DEV-5312] Generalize event subscribers and dispatchers 2019-11-25 12:42:03 -05:00
Austin Schaffer e781a841b1 [DEV-5312] Reconnect AMQP when connection drops 2019-11-25 12:26:39 -05:00
Austin Schaffer faa7e15760 [DEV-5312] Add signal handling and prometheus monitoring 2019-11-22 16:37:57 -05:00
Austin Schaffer 9b5cd4e89f [DEV-5312] Call data-processor and instantiate classes 2019-11-20 09:48:28 -05:00
Austin Schaffer c5733d1dff [DEV-5312] Add interface for amqp publisher and implement a delta publisher 2019-11-12 17:04:54 -05:00
Austin Schaffer 950ae8818b [DEV-5312] Add preliminary processor 2019-11-12 16:21:01 -05:00
Austin Schaffer 91a7cf94b2 [DEV-5312] Add dao for deltas 2019-11-12 16:21:01 -05:00
Mike Gerwitz d0b2a4ce73 bin/server: Convert to TypeScript 2019-11-12 16:21:00 -05:00
Austin Schaffer 015a3f5d47 [DEV-5312] Convert MongoServerDao to TS 2019-11-11 15:14:49 -05:00
Austin Schaffer c2458dee78 [DEV-5312] Convert most parts of RatingService Publish to TS 2019-11-11 09:29:07 -05:00
Austin Schaffer cda048dd8c [DEV-6353] Generate and save reverse deltas for ratingdata and step save 2019-11-07 16:25:33 -05:00
Austin Schaffer a4d151ef79 [DEV-6353] Add function to create delta from one data set to another
The type of the delta must accomodate undefined and null values for array
data, as well as null for a key value to represent removing a key
2019-11-07 16:25:33 -05:00
Austin Schaffer 26c4367ead [DEV-6353] Accept push data in MongoDao 2019-11-07 16:25:33 -05:00
Austin Schaffer 48aa315c2a [DEV-6353] Get and set past rate data 2019-11-07 16:25:33 -05:00
Austin Schaffer 4f39c754a2 [DEV-6353] Add bucket definitions 2019-11-07 16:25:33 -05:00
Austin Schaffer 6fcf72f1b6 [DEV-6353] Convert DataProcessor to TS 2019-11-07 16:25:33 -05:00
Mike Gerwitz 64718d5513 RatingService: Fix initialRatedDate and lastRatedDate reply
Recent changes caused the quote to be read before it was mutated by
_saveRatingData.
2019-10-31 14:26:12 -04:00
Mike Gerwitz f0e246deef RatingService#postProcessRaterData: Fix accidental indvRate change
I accidentally changed id=>after when prototyping data for the _new_
deferred rating (not yet written).  There were no tests for this code, which
was originally written in July of 2012 (see private lovullo.git repo), so it
was not caught until QA.
2019-10-31 11:21:01 -04:00
Mike Gerwitz 65e7880c81 RatingService: Improved error handling
This does only a slightly better job than before.
2019-10-29 13:36:55 -04:00
Mike Gerwitz 1ed8ad1cd7 RatingService: Return promise
This only replaces the callbacks at the highest level and starts to move
toward proper error handling.  The private methods do not yet properly
propagate errors.
2019-10-29 13:36:55 -04:00
Mike Gerwitz 1aa69c2a56 RatingService: Save rating data to separate field (ratedata)
This adds ratedata to the quote collection.  We use the existing saveQuote
method so that this operation is atomic.  We're also continuing to save to
the bucket for now so as not to break existing code, but the intent in the
future will be to remove all but necessary data that should be exposed to
the client.
2019-10-29 13:36:55 -04:00
Mike Gerwitz cffd9ddeb0 RatingServiceTest: Add type information for stubs
This also gets rid of the RatingServiceStub module, which is not used by
anything else.  I suspect that I originally added it to be shared by traits,
but that's no longer going to be the case (and the only remaining trait is
unfortunately untested atm, and will be going away).
2019-10-29 13:36:55 -04:00
Mike Gerwitz 767a248e44 RatingService: Convert to TypeScript
This was an adventure, and was also used as a peer programming exercise to
introduce TypeScript to other programmers in the office.

This class has far too many dependencies, which made this difficult.  The
approach was to create .d.ts files for dependencies and wait on moving those
over for now, otherwise the task will never get done.

The RatingServicePublic trait was left as such for the time being; I was
able to work around a bug that was making it difficult to mix it into a
prototype.

There were no logic changes; this was just type refactoring.
2019-10-29 13:36:41 -04:00
Mike Gerwitz e2edbfc7b2 src/numeric: New module
This beings to introduce compile-time safety for numeric values under the
assumption that they are enforced by the runtime.  See docblock for more
information.
2019-10-24 10:29:33 -04:00
Mike Gerwitz 7583cc1a71 RatingServiceSubmitNotify: Remove unused trait 2019-10-21 14:06:47 -04:00
Mike Gerwitz 44ad6437e2 {src,test}/.npmignore: New files
NPM was not publishing the compiled JS files (from the TS input) because
they were present in generated .gitignore files.  .npmignore takes
precedence.
2019-10-21 13:22:24 -04:00
Mike Gerwitz 16409a014f TokenedDataApi: Accept or kill token
Rather than leaving a token in a DONE state, we should either transition to
ACCEPTED or DEAD depending on whether the token was superceded.

* src/server/dapi/TokenedDataApi.ts (_replyUnlessStale): Accept token when
  not superceded, otherwise kill.
  [store]: New param.
  (request): Use it.
* test/server/dapi/TokenedDataApiTest.ts: Update accordingly.
2019-10-18 09:55:11 -04:00
Mike Gerwitz e003abcd4b Integrate TokenedDataApi into system
This changes the easejs interface for DataApi, which requires adding the
param to everything.  The TS interface was created in a previous commit and
already contained this parameter.  The idea is to remove the easejs
interface in the future, but traits are a barrier to that atm.

DocumentServer and controller demonstrate the mess that we have with regards
to instantiating dependencies.  This needs to change---DocumentServer itself
was something that was started but never fully realized.  It makes this
incredibly confusing, difficult to follow, and complicates important error
handling that ought to be taking place.  It also discourages implementing
additional dependencies.

I'm not going to go through and provide a ChangeLog-style commit message for
this commit.  I'm too exhausted by this crap.
2019-10-18 09:55:11 -04:00
Mike Gerwitz d8c065817f NodeCallback<T, R>: New type to simplify callback declarations
Just trying to reduce some boilerplate.  I kept this as a separate commit to
illustrate clearly how this type of things is done since we'll have people
learning TypeScript.

* src/types/misc.ts (NodeCallback<T,R>): New type.
* src/server/dapi/TokenedDataApi.ts: Use it.
* test/server/dapi/TokenedDataApiTest.ts: Use it.
2019-10-18 09:55:11 -04:00