1
0
Fork 0
Commit Graph

447 Commits (103c06da3938a78a1badc35a051fad4998ac7274)

Author SHA1 Message Date
Austin Schaffer 103c06da39 [DEV-6730] Post whether or not a save step will conclude the process 2020-01-15 10:26:07 -05:00
Schaffer, Austin aea913e767 [DEV-5314] Do not publish directly to postrate exchange after rating 2020-01-02 16:42:25 -05:00
Austin Schaffer 3fbe9b31ff [DEV-6763] Include ratedata in data deltas 2019-12-26 15:28:46 -05:00
Joseph Frazer aa2ba885af [DEV-6756] Prevent special chars from "local-part" of email address
The `"(),:;<>@[\]`` are allowed in the "local-part" of an email address
under certain circumstances, but our regex is not checking those
circumstances. It is better if we do not allow them until we improve
validation.
2019-12-23 08:26:54 -05:00
Austin Schaffer 9f22db72fd [DEV-5312] Use milliseconds for timestamp 2019-12-19 14:42:10 -05:00
Austin Schaffer e2a9d17dc6 [DEV-5312] Do not use Object.create when applying delta 2019-12-19 14:42:10 -05:00
Austin Schaffer 1cb47d4acf [DEV-6710] [WIP] Save environment on quote save 2019-12-19 14:17:06 -05:00
Austin Schaffer 5e5932fe9d [DEV-5312] Get delta processor program field from mongodb 2019-12-18 11:36:40 -05:00
Austin Schaffer d7f01e12c8 [DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores 2019-12-18 09:48:26 -05:00
Austin Schaffer b9f05b174f [DEV-5312] Update avro schema error context so invalid path is not overwritten 2019-12-18 09:43:14 -05:00
Austin Schaffer 699ee29a95 [DEV-5312] Use formatted time when logging events 2019-12-17 11:59:06 -05:00
Austin Schaffer 542745da8a [DEV-5312] Log quote_id along with doc_id 2019-12-17 11:59:06 -05:00
Austin Schaffer ccfd40c64f [DEV-5312] Allow for arbitrary depth in bucket schema 2019-12-17 11:59:01 -05:00
Austin Schaffer b6f72b4b09 [DEV-5312] Create additional indexes and process less frequently 2019-12-16 09:27:29 -05:00
Austin Schaffer cf0ca51cef [DEV-5312] Include created and modified fields in avro schema 2019-12-12 14:31:12 -05:00
Austin Schaffer e3dded760d [DEV-5312] Define document meta data and restructure message writer.
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9eb1f3afca DeltaProcessor: Correct delta application
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 819701eca3 DeltaProcessor: Encapsulate getDeltas and remove redundant tests
getDeltas should be encapsulated.  It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz d0a3ba573f DeltaProcessor: Remove redundant tests
These are covered by other tests and are testing too closely to the
implementation.  This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 8e0c0e22a8 DeltaPublisher#_sendMessage: Promise.reject=>throw
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz fb80d2a8b4 .env: Use uppercase for env vars 2019-12-12 10:27:09 -05:00
Mike Gerwitz 84c2601f1c EventMediator: Log error stack under context 2019-12-12 10:27:09 -05:00
Mike Gerwitz e885026e0a DeltaPublisher: Remove parsing from constructor 2019-12-12 10:27:09 -05:00
Mike Gerwitz 4383d15c5a AvroFactory: Extract avro-js type definitions into src/types
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9d6cb23e16 AmqpConnection: Propagate assertExchange promise
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer d9ee999adb [DEV-5312] Refactor class dependencies
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer 1ddb6f29eb [DEV-5312] Add logic to apply delta to bucket then publish modified bucket 2019-12-12 10:13:27 -05:00
Austin Schaffer 1b96cd9147 [DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus 2019-12-12 10:10:45 -05:00
Austin Schaffer 5ee9a5d340 [DEV-5312] Generalize event subscribers and dispatchers 2019-11-25 12:42:03 -05:00
Austin Schaffer e781a841b1 [DEV-5312] Reconnect AMQP when connection drops 2019-11-25 12:26:39 -05:00
Austin Schaffer faa7e15760 [DEV-5312] Add signal handling and prometheus monitoring 2019-11-22 16:37:57 -05:00
Austin Schaffer 9b5cd4e89f [DEV-5312] Call data-processor and instantiate classes 2019-11-20 09:48:28 -05:00
Austin Schaffer 309585cf6e [DEV-5312] Set processed flag 2019-11-13 12:04:27 -05:00
Austin Schaffer e737850efd [DEV-5312] Set processed flag 2019-11-12 17:10:19 -05:00
Austin Schaffer c5733d1dff [DEV-5312] Add interface for amqp publisher and implement a delta publisher 2019-11-12 17:04:54 -05:00
Burzlaff, Herbert 341aeb6a08 [DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead 2019-11-12 16:21:01 -05:00
Austin Schaffer 950ae8818b [DEV-5312] Add preliminary processor 2019-11-12 16:21:01 -05:00
Austin Schaffer 91a7cf94b2 [DEV-5312] Add dao for deltas 2019-11-12 16:21:01 -05:00
Mike Gerwitz d0b2a4ce73 bin/server: Convert to TypeScript 2019-11-12 16:21:00 -05:00
Austin Schaffer bdd77f8de4 [DEV-5312] Workaround multiple easejs instances 2019-11-11 15:15:28 -05:00
Austin Schaffer 015a3f5d47 [DEV-5312] Convert MongoServerDao to TS 2019-11-11 15:14:49 -05:00
Austin Schaffer c2458dee78 [DEV-5312] Convert most parts of RatingService Publish to TS 2019-11-11 09:29:07 -05:00
Mike Gerwitz 1e5a0fb701 events: Use EventEmitter from old version of Node
This was copied from a _very_ early version of node.  The problem is that,
for reasons not yet investigated, Node v12's EventEmitter (at least when
extended by easejs) is causing data to be shared across instances.

This kicks the can down the road a little bit.  The goal is to eventually
get rid of EventEmitter entirely.

Here's the script used to do the conversion in this commit:

git grep -l "require( 'events' )" -- src \
  | while read f; do
      path=$( sed 's|src/||;s|[^/]\+|..|g' <<< "$(dirname "$f")" )
      sed -i "s|require( 'events' )|require( '$path/events' )|" "$f"
    done
2019-11-08 13:06:18 -05:00
Austin Schaffer cda048dd8c [DEV-6353] Generate and save reverse deltas for ratingdata and step save 2019-11-07 16:25:33 -05:00
Austin Schaffer a4d151ef79 [DEV-6353] Add function to create delta from one data set to another
The type of the delta must accomodate undefined and null values for array
data, as well as null for a key value to represent removing a key
2019-11-07 16:25:33 -05:00
Austin Schaffer 26c4367ead [DEV-6353] Accept push data in MongoDao 2019-11-07 16:25:33 -05:00
Austin Schaffer 48aa315c2a [DEV-6353] Get and set past rate data 2019-11-07 16:25:33 -05:00
Austin Schaffer 9f8fb90d0d [DEV-6353] Add dapi definitions 2019-11-07 16:25:33 -05:00
Austin Schaffer 4f39c754a2 [DEV-6353] Add bucket definitions 2019-11-07 16:25:33 -05:00
Austin Schaffer 6fcf72f1b6 [DEV-6353] Convert DataProcessor to TS 2019-11-07 16:25:33 -05:00