james_woods
Joined 11 years ago
Joined 11 years ago
>When Ray and I were designing Sequel in 1974, we thought that the predominant use of the language would be for ad-hoc queries by planners and other professionals whose domain of expertise was not primarily data- base management. We wanted the language to be simple enough that ordinary people could ‘‘walk up and use it’’ with a minimum of training.
What I would like to have is a choice - and Apache Beam for example lets you choose this.
I know how long I want to wait, 30 minutes in one of my cases as I know that I've seen 95% of the important data by then. In the streaming world there is _always_ late data so being able to tell what should happen when the rest (5%) arrives is crucial for me.
This differs from use-case to use-case for me and being able to configure this and handling out-of-order data at scale is key for me when selecting a framework for stream processing. Apache Beam and Apache Flink do this very well.
Taken from [1]: Apache Beam has some other approach where you use both and there is some magical timeout and it only works for windows or something and blah blah blah... If any of you all know the details, drop me a note. It obviously only works when you window your data as it needs to fit in memory. The event-time and system-time concept from Beam and Flink are very similar, also the watermark approach. Thank you for sharing the links, For me it is now clearer where the difference lies between differential-dataflow and stream-processing frameworks (which also offer SQL and even ACID conformity!). I'm using Beam/Flink in production and missing out on one of these mentioned points is a deal-breaker for me.
https://www.oreilly.com/radar/the-world-beyond-batch-streami...
https://www.oreilly.com/radar/the-world-beyond-batch-streami...
Also "Materialize" seems not to support needed features like tumbling windows (yet) when dealing with streaming data in SQL: https://arxiv.org/abs/1905.12133
Additionally "Materialize" states in their doc: State is all in totally volatile memory; if materialized dies, so too does all of the data. - this is not true for example for Apache Flink which stores its state in systems like RocksDB.
Having SideInputs or seeds is pretty neat, imagine you have two tables of several TiBs or larger. This is also something that "Materialize" currently lacks: Streaming sources must receive all of their data from the stream itself; there is no way to “seed” a streaming source with static data.
Ah - they persuaded them. What a lovely euphemism.
"But there’s no clear evidence that these mandates improve financial performance. A recent study of S&P 500 companies that was conducted by University of Pittsburgh researchers found that executives are “using RTO mandates to reassert control over employees and blame employees as a scapegoat for bad firm performance.” Those policies result in “significant declines in employees’ job satisfaction but no significant changes in financial performance or firm values,” they concluded." (https://sloanreview.mit.edu/article/return-to-office-mandate...)