5e5932fe9d
[DEV-5312] Get delta processor program field from mongodb
2019-12-18 11:36:40 -05:00
d7f01e12c8
[DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores
2019-12-18 09:48:26 -05:00
b9f05b174f
[DEV-5312] Update avro schema error context so invalid path is not overwritten
2019-12-18 09:43:14 -05:00
c64ecbc816
[DEV-5312] Update the avro schema to accept arbitrary array depths
2019-12-17 13:20:20 -05:00
699ee29a95
[DEV-5312] Use formatted time when logging events
2019-12-17 11:59:06 -05:00
542745da8a
[DEV-5312] Log quote_id along with doc_id
2019-12-17 11:59:06 -05:00
ccfd40c64f
[DEV-5312] Allow for arbitrary depth in bucket schema
2019-12-17 11:59:01 -05:00
33f39d5760
Use milliseconds for timestamps and mongo index creation
2019-12-16 09:41:44 -05:00
b6f72b4b09
[DEV-5312] Create additional indexes and process less frequently
2019-12-16 09:27:29 -05:00
abf76beac6
Update avro and prometheus dependencies
2019-12-13 09:50:50 -05:00
b93188c578
Add bin/delta-processor to EXTRA_DIST in Makefile
2019-12-12 16:03:50 -05:00
6c38f7d164
Merge branch 'jira-5312' into 'master'
...
[DEV-5312] Process to publish quote modifications to new exchange
See merge request floss/liza!68
2019-12-12 15:21:04 -05:00
cf0ca51cef
[DEV-5312] Include created and modified fields in avro schema
2019-12-12 14:31:12 -05:00
ad9e66c83e
[DEV-5312] Add delta-processor to bin
2019-12-12 12:38:17 -05:00
e3dded760d
[DEV-5312] Define document meta data and restructure message writer.
...
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9eb1f3afca
DeltaProcessor: Correct delta application
...
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
819701eca3
DeltaProcessor: Encapsulate getDeltas and remove redundant tests
...
getDeltas should be encapsulated. It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
d0a3ba573f
DeltaProcessor: Remove redundant tests
...
These are covered by other tests and are testing too closely to the
implementation. This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
8e0c0e22a8
DeltaPublisher#_sendMessage: Promise.reject=>throw
...
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
fb80d2a8b4
.env: Use uppercase for env vars
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e13a062f2b
bin/delta-processor: Log DeltaProcessor#process errors
2019-12-12 10:27:09 -05:00
Mike Gerwitz
2fb47cc993
bin/delta-processor: Provide error to EventMediator
...
This will log in the same format as the rest of the system.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
84c2601f1c
EventMediator: Log error stack under context
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e885026e0a
DeltaPublisher: Remove parsing from constructor
2019-12-12 10:27:09 -05:00
Mike Gerwitz
4383d15c5a
AvroFactory: Extract avro-js type definitions into src/types
...
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9d6cb23e16
AmqpConnection: Propagate assertExchange promise
...
Failures were being ignored.
2019-12-12 10:27:09 -05:00
d9ee999adb
[DEV-5312] Refactor class dependencies
...
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
1ddb6f29eb
[DEV-5312] Add logic to apply delta to bucket then publish modified bucket
2019-12-12 10:13:27 -05:00
1b96cd9147
[DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus
2019-12-12 10:10:45 -05:00
5ee9a5d340
[DEV-5312] Generalize event subscribers and dispatchers
2019-11-25 12:42:03 -05:00
e781a841b1
[DEV-5312] Reconnect AMQP when connection drops
2019-11-25 12:26:39 -05:00
faa7e15760
[DEV-5312] Add signal handling and prometheus monitoring
2019-11-22 16:37:57 -05:00
9b5cd4e89f
[DEV-5312] Call data-processor and instantiate classes
2019-11-20 09:48:28 -05:00
309585cf6e
[DEV-5312] Set processed flag
2019-11-13 12:04:27 -05:00
e737850efd
[DEV-5312] Set processed flag
2019-11-12 17:10:19 -05:00
c5733d1dff
[DEV-5312] Add interface for amqp publisher and implement a delta publisher
2019-11-12 17:04:54 -05:00
Mike Gerwitz
de94f69e8f
bin/server.ts: fs.unlink requires two arguments on Node v12
2019-11-12 16:21:01 -05:00
341aeb6a08
[DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead
2019-11-12 16:21:01 -05:00
950ae8818b
[DEV-5312] Add preliminary processor
2019-11-12 16:21:01 -05:00
91a7cf94b2
[DEV-5312] Add dao for deltas
2019-11-12 16:21:01 -05:00
Mike Gerwitz
d0b2a4ce73
bin/server: Convert to TypeScript
2019-11-12 16:21:00 -05:00
Mike Gerwitz
e450bbb3d6
bin/delta-processor: Placeholder script
2019-11-11 16:53:58 -05:00
bdd77f8de4
[DEV-5312] Workaround multiple easejs instances
2019-11-11 15:15:28 -05:00
015a3f5d47
[DEV-5312] Convert MongoServerDao to TS
2019-11-11 15:14:49 -05:00
c2458dee78
[DEV-5312] Convert most parts of RatingService Publish to TS
2019-11-11 09:29:07 -05:00
Mike Gerwitz
31a820a34f
Mitigate problems caused by changes in EventEmitter
...
This kicks the can down the road a bit since we do not have time atm the
investigate what changes in EventEmitter are causing the underlying
problem. With that said, the hope is to get rid of EventEmitter entirely.
DEV-6488
2019-11-11 09:16:29 -05:00
Mike Gerwitz
1e5a0fb701
events: Use EventEmitter from old version of Node
...
This was copied from a _very_ early version of node. The problem is that,
for reasons not yet investigated, Node v12's EventEmitter (at least when
extended by easejs) is causing data to be shared across instances.
This kicks the can down the road a little bit. The goal is to eventually
get rid of EventEmitter entirely.
Here's the script used to do the conversion in this commit:
git grep -l "require( 'events' )" -- src \
| while read f; do
path=$( sed 's|src/||;s|[^/]\+|..|g' <<< "$(dirname "$f")" )
sed -i "s|require( 'events' )|require( '$path/events' )|" "$f"
done
2019-11-08 13:06:18 -05:00
Mike Gerwitz
11c2bcb9b1
configure (TS_NODE_VERSION): Make configurable
...
Apparently @types/node is not always available for the current node version...
2019-11-08 13:05:34 -05:00
Mike Gerwitz
c8c7dfda5d
Makefile.am (check-ts-out): New check for TS->JS compilation
...
tsc handles its own incremental builds, and if a file is removed, it isn't
always regenerated. This resulted in a bad distribution being generated and
published to npm.
2019-11-08 11:26:35 -05:00
6cc72d9d11
[DEV-6353] Generate and store reverse delta on step save
...
Co-Authored-By: Mike Gerwitz <mike.gerwitz@ryansg.com>
See merge request floss/liza!66
2019-11-08 08:41:37 -05:00
Austin Schaffer
Mike Gerwitz