Where do I put X data in MindFlayer
Normal, Everyday, Data we don't want to lose: the Event Store (Postgres right now)
The core things like api tokens, configuration data, etc... this should all be stored in the event store so that we can use it in any way we sit fit in the future. We gain great reporting, ability to stream changes, and project into convenient read formats.
Realtime data: Kafka
When it comes to analytics and realtime reporting we want to put it on Kafka. This will eventually be picked up by our Flink job and transformed into various reporting forms (Domo at the present) and archiving (s3) stores.
Caching / Read Friendly Store: Redis
Redis is the primary database that we read stuff out of because it's very fast to store and read from. Mind you, it's an in memory store so don't store anything there you can't afford to loose.