That’s not how stream processing works? It’s absurdly costly to query snowflake for every record that streams in. So then you try to cache the snowflake dataset. But then the cache becomes stale. Honestly this sounds like an XY problem — the problem is stated as X but you’re really trying to solve something else Y.
What is the business requirement? What are the technical specifications required to meet the need? From there, we can start to consider architecture solutions.
Storage is relatively cheap compared to compute. Most stream processors require or at least highly encourage you to provide them direct access to the input data instead of calling an external system.
What is the business requirement? What are the technical specifications required to meet the need? From there, we can start to consider architecture solutions.
Storage is relatively cheap compared to compute. Most stream processors require or at least highly encourage you to provide them direct access to the input data instead of calling an external system.