1
0
Fork 0
Commit Graph

739 Commits (b9f05b174fdb8da72eadcaa2f5fe978acae2ef33)

Author SHA1 Message Date
Austin Schaffer b9f05b174f [DEV-5312] Update avro schema error context so invalid path is not overwritten 2019-12-18 09:43:14 -05:00
Austin Schaffer c64ecbc816 [DEV-5312] Update the avro schema to accept arbitrary array depths 2019-12-17 13:20:20 -05:00
Austin Schaffer 699ee29a95 [DEV-5312] Use formatted time when logging events 2019-12-17 11:59:06 -05:00
Austin Schaffer 542745da8a [DEV-5312] Log quote_id along with doc_id 2019-12-17 11:59:06 -05:00
Austin Schaffer ccfd40c64f [DEV-5312] Allow for arbitrary depth in bucket schema 2019-12-17 11:59:01 -05:00
Austin Schaffer 33f39d5760 Use milliseconds for timestamps and mongo index creation 2019-12-16 09:41:44 -05:00
Austin Schaffer b6f72b4b09 [DEV-5312] Create additional indexes and process less frequently 2019-12-16 09:27:29 -05:00
Austin Schaffer abf76beac6 Update avro and prometheus dependencies 2019-12-13 09:50:50 -05:00
Austin Schaffer b93188c578 Add bin/delta-processor to EXTRA_DIST in Makefile 2019-12-12 16:03:50 -05:00
Schaffer, Austin 6c38f7d164 Merge branch 'jira-5312' into 'master'
[DEV-5312] Process to publish quote modifications to new exchange

See merge request floss/liza!68
2019-12-12 15:21:04 -05:00
Austin Schaffer cf0ca51cef [DEV-5312] Include created and modified fields in avro schema 2019-12-12 14:31:12 -05:00
Austin Schaffer ad9e66c83e [DEV-5312] Add delta-processor to bin 2019-12-12 12:38:17 -05:00
Austin Schaffer e3dded760d [DEV-5312] Define document meta data and restructure message writer.
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9eb1f3afca DeltaProcessor: Correct delta application
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 819701eca3 DeltaProcessor: Encapsulate getDeltas and remove redundant tests
getDeltas should be encapsulated.  It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz d0a3ba573f DeltaProcessor: Remove redundant tests
These are covered by other tests and are testing too closely to the
implementation.  This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 8e0c0e22a8 DeltaPublisher#_sendMessage: Promise.reject=>throw
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz fb80d2a8b4 .env: Use uppercase for env vars 2019-12-12 10:27:09 -05:00
Mike Gerwitz e13a062f2b bin/delta-processor: Log DeltaProcessor#process errors 2019-12-12 10:27:09 -05:00
Mike Gerwitz 2fb47cc993 bin/delta-processor: Provide error to EventMediator
This will log in the same format as the rest of the system.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 84c2601f1c EventMediator: Log error stack under context 2019-12-12 10:27:09 -05:00
Mike Gerwitz e885026e0a DeltaPublisher: Remove parsing from constructor 2019-12-12 10:27:09 -05:00
Mike Gerwitz 4383d15c5a AvroFactory: Extract avro-js type definitions into src/types
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9d6cb23e16 AmqpConnection: Propagate assertExchange promise
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer d9ee999adb [DEV-5312] Refactor class dependencies
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer 1ddb6f29eb [DEV-5312] Add logic to apply delta to bucket then publish modified bucket 2019-12-12 10:13:27 -05:00
Austin Schaffer 1b96cd9147 [DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus 2019-12-12 10:10:45 -05:00
Austin Schaffer 5ee9a5d340 [DEV-5312] Generalize event subscribers and dispatchers 2019-11-25 12:42:03 -05:00
Austin Schaffer e781a841b1 [DEV-5312] Reconnect AMQP when connection drops 2019-11-25 12:26:39 -05:00
Austin Schaffer faa7e15760 [DEV-5312] Add signal handling and prometheus monitoring 2019-11-22 16:37:57 -05:00
Austin Schaffer 9b5cd4e89f [DEV-5312] Call data-processor and instantiate classes 2019-11-20 09:48:28 -05:00
Austin Schaffer 309585cf6e [DEV-5312] Set processed flag 2019-11-13 12:04:27 -05:00
Austin Schaffer e737850efd [DEV-5312] Set processed flag 2019-11-12 17:10:19 -05:00
Austin Schaffer c5733d1dff [DEV-5312] Add interface for amqp publisher and implement a delta publisher 2019-11-12 17:04:54 -05:00
Mike Gerwitz de94f69e8f bin/server.ts: fs.unlink requires two arguments on Node v12 2019-11-12 16:21:01 -05:00
Burzlaff, Herbert 341aeb6a08 [DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead 2019-11-12 16:21:01 -05:00
Austin Schaffer 950ae8818b [DEV-5312] Add preliminary processor 2019-11-12 16:21:01 -05:00
Austin Schaffer 91a7cf94b2 [DEV-5312] Add dao for deltas 2019-11-12 16:21:01 -05:00
Mike Gerwitz d0b2a4ce73 bin/server: Convert to TypeScript 2019-11-12 16:21:00 -05:00
Mike Gerwitz e450bbb3d6 bin/delta-processor: Placeholder script 2019-11-11 16:53:58 -05:00
Austin Schaffer bdd77f8de4 [DEV-5312] Workaround multiple easejs instances 2019-11-11 15:15:28 -05:00
Austin Schaffer 015a3f5d47 [DEV-5312] Convert MongoServerDao to TS 2019-11-11 15:14:49 -05:00
Austin Schaffer c2458dee78 [DEV-5312] Convert most parts of RatingService Publish to TS 2019-11-11 09:29:07 -05:00
Mike Gerwitz 31a820a34f Mitigate problems caused by changes in EventEmitter
This kicks the can down the road a bit since we do not have time atm the
investigate what changes in EventEmitter are causing the underlying
problem.  With that said, the hope is to get rid of EventEmitter entirely.

DEV-6488
2019-11-11 09:16:29 -05:00
Mike Gerwitz 1e5a0fb701 events: Use EventEmitter from old version of Node
This was copied from a _very_ early version of node.  The problem is that,
for reasons not yet investigated, Node v12's EventEmitter (at least when
extended by easejs) is causing data to be shared across instances.

This kicks the can down the road a little bit.  The goal is to eventually
get rid of EventEmitter entirely.

Here's the script used to do the conversion in this commit:

git grep -l "require( 'events' )" -- src \
  | while read f; do
      path=$( sed 's|src/||;s|[^/]\+|..|g' <<< "$(dirname "$f")" )
      sed -i "s|require( 'events' )|require( '$path/events' )|" "$f"
    done
2019-11-08 13:06:18 -05:00
Mike Gerwitz 11c2bcb9b1 configure (TS_NODE_VERSION): Make configurable
Apparently @types/node is not always available for the current node version...
2019-11-08 13:05:34 -05:00
Mike Gerwitz c8c7dfda5d Makefile.am (check-ts-out): New check for TS->JS compilation
tsc handles its own incremental builds, and if a file is removed, it isn't
always regenerated.  This resulted in a bad distribution being generated and
published to npm.
2019-11-08 11:26:35 -05:00
Schaffer, Austin 6cc72d9d11 [DEV-6353] Generate and store reverse delta on step save
Co-Authored-By: Mike Gerwitz <mike.gerwitz@ryansg.com>

See merge request floss/liza!66
2019-11-08 08:41:37 -05:00
Austin Schaffer cda048dd8c [DEV-6353] Generate and save reverse deltas for ratingdata and step save 2019-11-07 16:25:33 -05:00
Austin Schaffer a4d151ef79 [DEV-6353] Add function to create delta from one data set to another
The type of the delta must accomodate undefined and null values for array
data, as well as null for a key value to represent removing a key
2019-11-07 16:25:33 -05:00