103c06da39
[DEV-6730] Post whether or not a save step will conclude the process
2020-01-15 10:26:07 -05:00
aea913e767
[DEV-5314] Do not publish directly to postrate exchange after rating
2020-01-02 16:42:25 -05:00
3fbe9b31ff
[DEV-6763] Include ratedata in data deltas
2019-12-26 15:28:46 -05:00
aa2ba885af
[DEV-6756] Prevent special chars from "local-part" of email address
...
The `"(),:;<>@[\]`` are allowed in the "local-part" of an email address
under certain circumstances, but our regex is not checking those
circumstances. It is better if we do not allow them until we improve
validation.
2019-12-23 08:26:54 -05:00
9f22db72fd
[DEV-5312] Use milliseconds for timestamp
2019-12-19 14:42:10 -05:00
e2a9d17dc6
[DEV-5312] Do not use Object.create when applying delta
2019-12-19 14:42:10 -05:00
1cb47d4acf
[DEV-6710] [WIP] Save environment on quote save
2019-12-19 14:17:06 -05:00
5e5932fe9d
[DEV-5312] Get delta processor program field from mongodb
2019-12-18 11:36:40 -05:00
d7f01e12c8
[DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores
2019-12-18 09:48:26 -05:00
b9f05b174f
[DEV-5312] Update avro schema error context so invalid path is not overwritten
2019-12-18 09:43:14 -05:00
699ee29a95
[DEV-5312] Use formatted time when logging events
2019-12-17 11:59:06 -05:00
542745da8a
[DEV-5312] Log quote_id along with doc_id
2019-12-17 11:59:06 -05:00
ccfd40c64f
[DEV-5312] Allow for arbitrary depth in bucket schema
2019-12-17 11:59:01 -05:00
b6f72b4b09
[DEV-5312] Create additional indexes and process less frequently
2019-12-16 09:27:29 -05:00
cf0ca51cef
[DEV-5312] Include created and modified fields in avro schema
2019-12-12 14:31:12 -05:00
e3dded760d
[DEV-5312] Define document meta data and restructure message writer.
...
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9eb1f3afca
DeltaProcessor: Correct delta application
...
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
819701eca3
DeltaProcessor: Encapsulate getDeltas and remove redundant tests
...
getDeltas should be encapsulated. It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
d0a3ba573f
DeltaProcessor: Remove redundant tests
...
These are covered by other tests and are testing too closely to the
implementation. This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
8e0c0e22a8
DeltaPublisher#_sendMessage: Promise.reject=>throw
...
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
fb80d2a8b4
.env: Use uppercase for env vars
2019-12-12 10:27:09 -05:00
Mike Gerwitz
84c2601f1c
EventMediator: Log error stack under context
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e885026e0a
DeltaPublisher: Remove parsing from constructor
2019-12-12 10:27:09 -05:00
Mike Gerwitz
4383d15c5a
AvroFactory: Extract avro-js type definitions into src/types
...
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9d6cb23e16
AmqpConnection: Propagate assertExchange promise
...
Failures were being ignored.
2019-12-12 10:27:09 -05:00
d9ee999adb
[DEV-5312] Refactor class dependencies
...
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
1ddb6f29eb
[DEV-5312] Add logic to apply delta to bucket then publish modified bucket
2019-12-12 10:13:27 -05:00
1b96cd9147
[DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus
2019-12-12 10:10:45 -05:00
5ee9a5d340
[DEV-5312] Generalize event subscribers and dispatchers
2019-11-25 12:42:03 -05:00
e781a841b1
[DEV-5312] Reconnect AMQP when connection drops
2019-11-25 12:26:39 -05:00
faa7e15760
[DEV-5312] Add signal handling and prometheus monitoring
2019-11-22 16:37:57 -05:00
9b5cd4e89f
[DEV-5312] Call data-processor and instantiate classes
2019-11-20 09:48:28 -05:00
309585cf6e
[DEV-5312] Set processed flag
2019-11-13 12:04:27 -05:00
e737850efd
[DEV-5312] Set processed flag
2019-11-12 17:10:19 -05:00
c5733d1dff
[DEV-5312] Add interface for amqp publisher and implement a delta publisher
2019-11-12 17:04:54 -05:00
341aeb6a08
[DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead
2019-11-12 16:21:01 -05:00
950ae8818b
[DEV-5312] Add preliminary processor
2019-11-12 16:21:01 -05:00
91a7cf94b2
[DEV-5312] Add dao for deltas
2019-11-12 16:21:01 -05:00
Mike Gerwitz
d0b2a4ce73
bin/server: Convert to TypeScript
2019-11-12 16:21:00 -05:00
bdd77f8de4
[DEV-5312] Workaround multiple easejs instances
2019-11-11 15:15:28 -05:00
015a3f5d47
[DEV-5312] Convert MongoServerDao to TS
2019-11-11 15:14:49 -05:00
c2458dee78
[DEV-5312] Convert most parts of RatingService Publish to TS
2019-11-11 09:29:07 -05:00
Mike Gerwitz
1e5a0fb701
events: Use EventEmitter from old version of Node
...
This was copied from a _very_ early version of node. The problem is that,
for reasons not yet investigated, Node v12's EventEmitter (at least when
extended by easejs) is causing data to be shared across instances.
This kicks the can down the road a little bit. The goal is to eventually
get rid of EventEmitter entirely.
Here's the script used to do the conversion in this commit:
git grep -l "require( 'events' )" -- src \
| while read f; do
path=$( sed 's|src/||;s|[^/]\+|..|g' <<< "$(dirname "$f")" )
sed -i "s|require( 'events' )|require( '$path/events' )|" "$f"
done
2019-11-08 13:06:18 -05:00
cda048dd8c
[DEV-6353] Generate and save reverse deltas for ratingdata and step save
2019-11-07 16:25:33 -05:00
a4d151ef79
[DEV-6353] Add function to create delta from one data set to another
...
The type of the delta must accomodate undefined and null values for array
data, as well as null for a key value to represent removing a key
2019-11-07 16:25:33 -05:00
26c4367ead
[DEV-6353] Accept push data in MongoDao
2019-11-07 16:25:33 -05:00
48aa315c2a
[DEV-6353] Get and set past rate data
2019-11-07 16:25:33 -05:00
9f8fb90d0d
[DEV-6353] Add dapi definitions
2019-11-07 16:25:33 -05:00
4f39c754a2
[DEV-6353] Add bucket definitions
2019-11-07 16:25:33 -05:00
6fcf72f1b6
[DEV-6353] Convert DataProcessor to TS
2019-11-07 16:25:33 -05:00
Austin Schaffer
Mike Gerwitz