Austin Schaffer
3fbe9b31ff
[DEV-6763] Include ratedata in data deltas
2019-12-26 15:28:46 -05:00
Joseph Frazer
8ae6e236e9
[DEV-6756] Prevent special chars from "local-part" of email address
...
Merge branch 'jira-6756'
* jira-6756:
[DEV-6756] Prevent special chars from "local-part" of email address
2019-12-23 08:42:48 -05:00
Joseph Frazer
aa2ba885af
[DEV-6756] Prevent special chars from "local-part" of email address
...
The `"(),:;<>@[\]`` are allowed in the "local-part" of an email address
under certain circumstances, but our regex is not checking those
circumstances. It is better if we do not allow them until we improve
validation.
2019-12-23 08:26:54 -05:00
Austin Schaffer
b48a9f0de2
Do not use Object.create when applying delta
2019-12-20 11:20:27 -05:00
Austin Schaffer
9f22db72fd
[DEV-5312] Use milliseconds for timestamp
2019-12-19 14:42:10 -05:00
Austin Schaffer
e2a9d17dc6
[DEV-5312] Do not use Object.create when applying delta
2019-12-19 14:42:10 -05:00
Schaffer, Austin
caf2c776b6
[DEV-6710] Introduce Concept of Environment to Quotes
2019-12-19 14:19:26 -05:00
Austin Schaffer
1cb47d4acf
[DEV-6710] [WIP] Save environment on quote save
2019-12-19 14:17:06 -05:00
Austin Schaffer
e819edc53b
[DEV-5312] Revert schema change; filter by double underscore; add programid; add step field on rate delta
2019-12-18 16:03:16 -05:00
Austin Schaffer
f4b4a9d5d9
[DEV-5312] Correct lowercase environment variable
2019-12-18 11:37:20 -05:00
Austin Schaffer
5e5932fe9d
[DEV-5312] Get delta processor program field from mongodb
2019-12-18 11:36:40 -05:00
Austin Schaffer
d7f01e12c8
[DEV-5312] Remove recursive avro schema and filter bucket keys starting with two underscores
2019-12-18 09:48:26 -05:00
Austin Schaffer
b9f05b174f
[DEV-5312] Update avro schema error context so invalid path is not overwritten
2019-12-18 09:43:14 -05:00
Austin Schaffer
c64ecbc816
[DEV-5312] Update the avro schema to accept arbitrary array depths
2019-12-17 13:20:20 -05:00
Austin Schaffer
699ee29a95
[DEV-5312] Use formatted time when logging events
2019-12-17 11:59:06 -05:00
Austin Schaffer
542745da8a
[DEV-5312] Log quote_id along with doc_id
2019-12-17 11:59:06 -05:00
Austin Schaffer
ccfd40c64f
[DEV-5312] Allow for arbitrary depth in bucket schema
2019-12-17 11:59:01 -05:00
Austin Schaffer
33f39d5760
Use milliseconds for timestamps and mongo index creation
2019-12-16 09:41:44 -05:00
Austin Schaffer
b6f72b4b09
[DEV-5312] Create additional indexes and process less frequently
2019-12-16 09:27:29 -05:00
Austin Schaffer
abf76beac6
Update avro and prometheus dependencies
2019-12-13 09:50:50 -05:00
Austin Schaffer
b93188c578
Add bin/delta-processor to EXTRA_DIST in Makefile
2019-12-12 16:03:50 -05:00
Schaffer, Austin
6c38f7d164
Merge branch 'jira-5312' into 'master'
...
[DEV-5312] Process to publish quote modifications to new exchange
See merge request floss/liza!68
2019-12-12 15:21:04 -05:00
Austin Schaffer
cf0ca51cef
[DEV-5312] Include created and modified fields in avro schema
2019-12-12 14:31:12 -05:00
Austin Schaffer
ad9e66c83e
[DEV-5312] Add delta-processor to bin
2019-12-12 12:38:17 -05:00
Austin Schaffer
e3dded760d
[DEV-5312] Define document meta data and restructure message writer.
...
Add better tests for message writer
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9eb1f3afca
DeltaProcessor: Correct delta application
...
Deltas must be applied in reverse to reproduce the state of the bucket or
rate data at that point in time, which is then paired with the delta that
will transform it into the previous state.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
819701eca3
DeltaProcessor: Encapsulate getDeltas and remove redundant tests
...
getDeltas should be encapsulated. It looks like it was public for the sake
of the tests, and its behavior can be inferred by looking at the result of
processing.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
d0a3ba573f
DeltaProcessor: Remove redundant tests
...
These are covered by other tests and are testing too closely to the
implementation. This should be encapsulated within the processor.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
8e0c0e22a8
DeltaPublisher#_sendMessage: Promise.reject=>throw
...
Purely stylistic.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
fb80d2a8b4
.env: Use uppercase for env vars
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e13a062f2b
bin/delta-processor: Log DeltaProcessor#process errors
2019-12-12 10:27:09 -05:00
Mike Gerwitz
2fb47cc993
bin/delta-processor: Provide error to EventMediator
...
This will log in the same format as the rest of the system.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
84c2601f1c
EventMediator: Log error stack under context
2019-12-12 10:27:09 -05:00
Mike Gerwitz
e885026e0a
DeltaPublisher: Remove parsing from constructor
2019-12-12 10:27:09 -05:00
Mike Gerwitz
4383d15c5a
AvroFactory: Extract avro-js type definitions into src/types
...
The declarations that were intended for the library were moved, but
additional things that were outside of the library were kept in place.
2019-12-12 10:27:09 -05:00
Mike Gerwitz
9d6cb23e16
AmqpConnection: Propagate assertExchange promise
...
Failures were being ignored.
2019-12-12 10:27:09 -05:00
Austin Schaffer
d9ee999adb
[DEV-5312] Refactor class dependencies
...
Move create config functions out of the main delta-processor.ts file. Change some any types to more specific types. Define document structure better. Move prometheus logic out of processor class. Do not call logger directly, listen for events and log them. Change logger to more PSR-3 compliant style. Extract amqp connection into its own class. Create avro and mongo objects in factory.
2019-12-12 10:14:54 -05:00
Austin Schaffer
1ddb6f29eb
[DEV-5312] Add logic to apply delta to bucket then publish modified bucket
2019-12-12 10:13:27 -05:00
Austin Schaffer
1b96cd9147
[DEV-5312] Refactor references from 'self' to 'this', pass console into event logger, and add factory for prometheus
2019-12-12 10:10:45 -05:00
Austin Schaffer
5ee9a5d340
[DEV-5312] Generalize event subscribers and dispatchers
2019-11-25 12:42:03 -05:00
Austin Schaffer
e781a841b1
[DEV-5312] Reconnect AMQP when connection drops
2019-11-25 12:26:39 -05:00
Austin Schaffer
faa7e15760
[DEV-5312] Add signal handling and prometheus monitoring
2019-11-22 16:37:57 -05:00
Austin Schaffer
9b5cd4e89f
[DEV-5312] Call data-processor and instantiate classes
2019-11-20 09:48:28 -05:00
Austin Schaffer
309585cf6e
[DEV-5312] Set processed flag
2019-11-13 12:04:27 -05:00
Austin Schaffer
e737850efd
[DEV-5312] Set processed flag
2019-11-12 17:10:19 -05:00
Austin Schaffer
c5733d1dff
[DEV-5312] Add interface for amqp publisher and implement a delta publisher
2019-11-12 17:04:54 -05:00
Mike Gerwitz
de94f69e8f
bin/server.ts: fs.unlink requires two arguments on Node v12
2019-11-12 16:21:01 -05:00
Burzlaff, Herbert
341aeb6a08
[DEVOPS-143] Node 12, needs a callback for asynchronous file writes, so switching to synchronous file writes instead
2019-11-12 16:21:01 -05:00
Austin Schaffer
950ae8818b
[DEV-5312] Add preliminary processor
2019-11-12 16:21:01 -05:00
Austin Schaffer
91a7cf94b2
[DEV-5312] Add dao for deltas
2019-11-12 16:21:01 -05:00