1
0
Fork 0
Commit Graph

748 Commits (b48a9f0de2e353adefb4f7968a250c0da5eff6c2)

Author SHA1 Message Date
Austin Schaffer b48a9f0de2 Do not use Object.create when applying delta 2019-12-20 11:20:27 -05:00
Austin Schaffer 9f22db72fd [DEV-5312] Use milliseconds for timestamp 2019-12-19 14:42:10 -05:00
Austin Schaffer e2a9d17dc6 [DEV-5312] Do not use Object.create when applying delta 2019-12-19 14:42:10 -05:00
Schaffer, Austin caf2c776b6 [DEV-6710] Introduce Concept of Environment to Quotes 2019-12-19 14:19:26 -05:00
Austin Schaffer 1cb47d4acf [DEV-6710] [WIP] Save environment on quote save 2019-12-19 14:17:06 -05:00
Austin Schaffer e819edc53b [DEV-5312] Revert schema change; filter by double underscore; add programid; add step field on rate delta 2019-12-18 16:03:16 -05:00
Austin Schaffer f4b4a9d5d9 [DEV-5312] Correct lowercase environment variable 2019-12-18 11:37:20 -05:00
Austin Schaffer 5e5932fe9d [DEV-5312] Get delta processor program field from mongodb 2019-12-18 11:36:40 -05:00
Austin Schaffer d7f01e12c8 [DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores 2019-12-18 09:48:26 -05:00
Austin Schaffer b9f05b174f [DEV-5312] Update avro schema error context so invalid path is not overwritten 2019-12-18 09:43:14 -05:00
Austin Schaffer c64ecbc816 [DEV-5312] Update the avro schema to accept arbitrary array depths 2019-12-17 13:20:20 -05:00
Austin Schaffer 699ee29a95 [DEV-5312] Use formatted time when logging events 2019-12-17 11:59:06 -05:00
Austin Schaffer 542745da8a [DEV-5312] Log quote_id along with doc_id 2019-12-17 11:59:06 -05:00
Austin Schaffer ccfd40c64f [DEV-5312] Allow for arbitrary depth in bucket schema 2019-12-17 11:59:01 -05:00
Austin Schaffer 33f39d5760 Use milliseconds for timestamps and mongo index creation 2019-12-16 09:41:44 -05:00
Austin Schaffer b6f72b4b09 [DEV-5312] Create additional indexes and process less frequently 2019-12-16 09:27:29 -05:00
Austin Schaffer abf76beac6 Update avro and prometheus dependencies 2019-12-13 09:50:50 -05:00
Austin Schaffer b93188c578 Add bin/delta-processor to EXTRA_DIST in Makefile 2019-12-12 16:03:50 -05:00
Schaffer, Austin 6c38f7d164 Merge branch 'jira-5312' into 'master'
[DEV-5312] Process to publish quote modifications to new exchange

See merge request floss/liza!68
2019-12-12 15:21:04 -05:00
Austin Schaffer cf0ca51cef [DEV-5312] Include created and modified fields in avro schema 2019-12-12 14:31:12 -05:00
Austin Schaffer ad9e66c83e [DEV-5312] Add delta-processor to bin 2019-12-12 12:38:17 -05:00
Austin Schaffer e3dded760d [DEV-5312] Define document meta data and restructure message writer.
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9eb1f3afca DeltaProcessor: Correct delta application
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 819701eca3 DeltaProcessor: Encapsulate getDeltas and remove redundant tests
getDeltas should be encapsulated.  It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz d0a3ba573f DeltaProcessor: Remove redundant tests
These are covered by other tests and are testing too closely to the
implementation.  This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 8e0c0e22a8 DeltaPublisher#_sendMessage: Promise.reject=>throw
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz fb80d2a8b4 .env: Use uppercase for env vars 2019-12-12 10:27:09 -05:00
Mike Gerwitz e13a062f2b bin/delta-processor: Log DeltaProcessor#process errors 2019-12-12 10:27:09 -05:00
Mike Gerwitz 2fb47cc993 bin/delta-processor: Provide error to EventMediator
This will log in the same format as the rest of the system.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 84c2601f1c EventMediator: Log error stack under context 2019-12-12 10:27:09 -05:00
Mike Gerwitz e885026e0a DeltaPublisher: Remove parsing from constructor 2019-12-12 10:27:09 -05:00
Mike Gerwitz 4383d15c5a AvroFactory: Extract avro-js type definitions into src/types
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz 9d6cb23e16 AmqpConnection: Propagate assertExchange promise
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer d9ee999adb [DEV-5312] Refactor class dependencies
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer 1ddb6f29eb [DEV-5312] Add logic to apply delta to bucket then publish modified bucket 2019-12-12 10:13:27 -05:00
Austin Schaffer 1b96cd9147 [DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus 2019-12-12 10:10:45 -05:00
Austin Schaffer 5ee9a5d340 [DEV-5312] Generalize event subscribers and dispatchers 2019-11-25 12:42:03 -05:00
Austin Schaffer e781a841b1 [DEV-5312] Reconnect AMQP when connection drops 2019-11-25 12:26:39 -05:00
Austin Schaffer faa7e15760 [DEV-5312] Add signal handling and prometheus monitoring 2019-11-22 16:37:57 -05:00
Austin Schaffer 9b5cd4e89f [DEV-5312] Call data-processor and instantiate classes 2019-11-20 09:48:28 -05:00
Austin Schaffer 309585cf6e [DEV-5312] Set processed flag 2019-11-13 12:04:27 -05:00
Austin Schaffer e737850efd [DEV-5312] Set processed flag 2019-11-12 17:10:19 -05:00
Austin Schaffer c5733d1dff [DEV-5312] Add interface for amqp publisher and implement a delta publisher 2019-11-12 17:04:54 -05:00
Mike Gerwitz de94f69e8f bin/server.ts: fs.unlink requires two arguments on Node v12 2019-11-12 16:21:01 -05:00
Burzlaff, Herbert 341aeb6a08 [DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead 2019-11-12 16:21:01 -05:00
Austin Schaffer 950ae8818b [DEV-5312] Add preliminary processor 2019-11-12 16:21:01 -05:00
Austin Schaffer 91a7cf94b2 [DEV-5312] Add dao for deltas 2019-11-12 16:21:01 -05:00
Mike Gerwitz d0b2a4ce73 bin/server: Convert to TypeScript 2019-11-12 16:21:00 -05:00
Mike Gerwitz e450bbb3d6 bin/delta-processor: Placeholder script 2019-11-11 16:53:58 -05:00
Austin Schaffer bdd77f8de4 [DEV-5312] Workaround multiple easejs instances 2019-11-11 15:15:28 -05:00