Austin Schaffer
b93188c578
Add bin/delta-processor to EXTRA_DIST in Makefile
2019-12-12 16:03:50 -05:00
Schaffer, Austin
6c38f7d164
Merge branch 'jira-5312' into 'master'
...
[DEV-5312] Process to publish quote modifications to new exchange
See merge request floss/liza!68
2019-12-12 15:21:04 -05:00
Austin Schaffer
cf0ca51cef
[DEV-5312] Include created and modified fields in avro schema
2019-12-12 14:31:12 -05:00
Austin Schaffer
ad9e66c83e
[DEV-5312] Add delta-processor to bin
2019-12-12 12:38:17 -05:00
Austin Schaffer
e3dded760d
[DEV-5312] Define document meta data and restructure message writer.
...
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9eb1f3afca
DeltaProcessor: Correct delta application
...
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
819701eca3
DeltaProcessor: Encapsulate getDeltas and remove redundant tests
...
getDeltas should be encapsulated. It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
d0a3ba573f
DeltaProcessor: Remove redundant tests
...
These are covered by other tests and are testing too closely to the
implementation. This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
8e0c0e22a8
DeltaPublisher#_sendMessage: Promise.reject=>throw
...
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
fb80d2a8b4
.env: Use uppercase for env vars
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e13a062f2b
bin/delta-processor: Log DeltaProcessor#process errors
2019-12-12 10:27:09 -05:00
Mike Gerwitz
2fb47cc993
bin/delta-processor: Provide error to EventMediator
...
This will log in the same format as the rest of the system.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
84c2601f1c
EventMediator: Log error stack under context
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e885026e0a
DeltaPublisher: Remove parsing from constructor
2019-12-12 10:27:09 -05:00
Mike Gerwitz
4383d15c5a
AvroFactory: Extract avro-js type definitions into src/types
...
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9d6cb23e16
AmqpConnection: Propagate assertExchange promise
...
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer
d9ee999adb
[DEV-5312] Refactor class dependencies
...
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer
1ddb6f29eb
[DEV-5312] Add logic to apply delta to bucket then publish modified bucket
2019-12-12 10:13:27 -05:00
Austin Schaffer
1b96cd9147
[DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus
2019-12-12 10:10:45 -05:00
Austin Schaffer
5ee9a5d340
[DEV-5312] Generalize event subscribers and dispatchers
2019-11-25 12:42:03 -05:00
Austin Schaffer
e781a841b1
[DEV-5312] Reconnect AMQP when connection drops
2019-11-25 12:26:39 -05:00
Austin Schaffer
faa7e15760
[DEV-5312] Add signal handling and prometheus monitoring
2019-11-22 16:37:57 -05:00
Austin Schaffer
9b5cd4e89f
[DEV-5312] Call data-processor and instantiate classes
2019-11-20 09:48:28 -05:00
Austin Schaffer
309585cf6e
[DEV-5312] Set processed flag
2019-11-13 12:04:27 -05:00
Austin Schaffer
e737850efd
[DEV-5312] Set processed flag
2019-11-12 17:10:19 -05:00
Austin Schaffer
c5733d1dff
[DEV-5312] Add interface for amqp publisher and implement a delta publisher
2019-11-12 17:04:54 -05:00
Mike Gerwitz
de94f69e8f
bin/server.ts: fs.unlink requires two arguments on Node v12
2019-11-12 16:21:01 -05:00
Burzlaff, Herbert
341aeb6a08
[DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead
2019-11-12 16:21:01 -05:00
Austin Schaffer
950ae8818b
[DEV-5312] Add preliminary processor
2019-11-12 16:21:01 -05:00
Austin Schaffer
91a7cf94b2
[DEV-5312] Add dao for deltas
2019-11-12 16:21:01 -05:00
Mike Gerwitz
d0b2a4ce73
bin/server: Convert to TypeScript
2019-11-12 16:21:00 -05:00
Mike Gerwitz
e450bbb3d6
bin/delta-processor: Placeholder script
2019-11-11 16:53:58 -05:00
Austin Schaffer
bdd77f8de4
[DEV-5312] Workaround multiple easejs instances
2019-11-11 15:15:28 -05:00
Austin Schaffer
015a3f5d47
[DEV-5312] Convert MongoServerDao to TS
2019-11-11 15:14:49 -05:00
Austin Schaffer
c2458dee78
[DEV-5312] Convert most parts of RatingService Publish to TS
2019-11-11 09:29:07 -05:00
Mike Gerwitz
31a820a34f
Mitigate problems caused by changes in EventEmitter
...
This kicks the can down the road a bit since we do not have time atm the
investigate what changes in EventEmitter are causing the underlying
problem. With that said, the hope is to get rid of EventEmitter entirely.
DEV-6488
2019-11-11 09:16:29 -05:00
Mike Gerwitz
1e5a0fb701
events: Use EventEmitter from old version of Node
...
This was copied from a _very_ early version of node. The problem is that,
for reasons not yet investigated, Node v12's EventEmitter (at least when
extended by easejs) is causing data to be shared across instances.
This kicks the can down the road a little bit. The goal is to eventually
get rid of EventEmitter entirely.
Here's the script used to do the conversion in this commit:
git grep -l "require( 'events' )" -- src \
| while read f; do
path=$( sed 's|src/||;s|[^/]\+|..|g' <<< "$(dirname "$f")" )
sed -i "s|require( 'events' )|require( '$path/events' )|" "$f"
done
2019-11-08 13:06:18 -05:00
Mike Gerwitz
11c2bcb9b1
configure (TS_NODE_VERSION): Make configurable
...
Apparently @types/node is not always available for the current node version...
2019-11-08 13:05:34 -05:00
Mike Gerwitz
c8c7dfda5d
Makefile.am (check-ts-out): New check for TS->JS compilation
...
tsc handles its own incremental builds, and if a file is removed, it isn't
always regenerated. This resulted in a bad distribution being generated and
published to npm.
2019-11-08 11:26:35 -05:00
Schaffer, Austin
6cc72d9d11
[DEV-6353] Generate and store reverse delta on step save
...
Co-Authored-By: Mike Gerwitz <mike.gerwitz@ryansg.com>
See merge request floss/liza!66
2019-11-08 08:41:37 -05:00
Austin Schaffer
cda048dd8c
[DEV-6353] Generate and save reverse deltas for ratingdata and step save
2019-11-07 16:25:33 -05:00
Austin Schaffer
a4d151ef79
[DEV-6353] Add function to create delta from one data set to another
...
The type of the delta must accomodate undefined and null values for array
data, as well as null for a key value to represent removing a key
2019-11-07 16:25:33 -05:00
Austin Schaffer
26c4367ead
[DEV-6353] Accept push data in MongoDao
2019-11-07 16:25:33 -05:00
Austin Schaffer
48aa315c2a
[DEV-6353] Get and set past rate data
2019-11-07 16:25:33 -05:00
Austin Schaffer
9f8fb90d0d
[DEV-6353] Add dapi definitions
2019-11-07 16:25:33 -05:00
Austin Schaffer
4f39c754a2
[DEV-6353] Add bucket definitions
2019-11-07 16:25:33 -05:00
Austin Schaffer
6fcf72f1b6
[DEV-6353] Convert DataProcessor to TS
2019-11-07 16:25:33 -05:00
Mike Gerwitz
2bdcacd11f
package.json.in: @types/node: Use installed node version
...
This will ensure that the right API version is installed.
2019-11-07 16:24:19 -05:00
Mike Gerwitz
365fe702cf
typescript v3.{6=>7} upgrade
2019-11-07 16:24:18 -05:00
Mike Gerwitz
64718d5513
RatingService: Fix initialRatedDate and lastRatedDate reply
...
Recent changes caused the quote to be read before it was mutated by
_saveRatingData.
2019-10-31 14:26:12 -04:00