itaifrenkel

The claim that Jews control or manipulate governments has been an antisemitic claim going back hundreds of years in multiple different variations. So, even if semetic does not equal Jew does not equal Zionist, any claim that one or the other controls a government requires some substance. Otherwise, it fits one of the most recognized template of antisemitic false claims.

Amazon is forcing elastic to make money from their hosted offering, instead of their proprietary x-pack modules. Professional Services don’t scale as software do (compare AWS with Rackspace) - so I’m ignoring that income stream for now. Basically, they are trying to force Elastic into the AWS red ocean. Elastic, in response, would have to make a technology leap in order to best solve their customers problems. Examples include: Integrated data prediction, anomaly detections, clustering (online machine learning). Seamless multi region cluster deployment (with some consistency guarantee)...

OP here. AMA

The problem is that OLAP is run by analysts/business/data.science people and they may mess up with their queries/workloads. without isolation the production performance could impact the user expirience

How would you run spark on top of the production db without affecting its performance?

The use case I'm referring to is when the client must be sharded to avoid CPU or network or disk bottlenecks.

The consumer groups proposal breaks the FIFO abstraction of a stream by allowing multiple clients to process a single stream.

Have you considered adding a semantic layer inside streams that allows each client to consume a substream? In effect the stream becomes multiplexed substreams.

If substreams makes the design too complex... have you considered server side stream 403 semantics? When a stream is manually deprecated it enters an immutable state and provides a redirect response with a link to another stream. This would allow multiplexing and demultiplexing streams without changing the client implementations too much.

For completeness I would state the obvious when fifo grouping is needed: 1. Scaling stateful event processing by splitting streams and adding more clients (CPU limit) 2. Scaling cross region replication by splitting streams and adding more tcp connections (network limit) 3. Handling more throughput by splitting a stream into two redis nodes (disk I/O limit)

Two comments on effectively once stream processing.

1. Consider adding an example for a stateful event stream processor client that saves the last read stream offset in redis, together with its current state and continues reading from that offset as an atomic operation. For example, a client that sums a stream of numbers, in order to have effectively once semantics would need to persist to redis the sum and offset together.

2. Consider adding a stream read deduplication example to mitigate clients that reinserted the same event twice. It is not clear how the client should behave if it didn't get an ack and it resents an event. What is the correct resending semantics so the reader would effectively dedup? What is the right data structure used to dedup message ids without consuming too much memory, etc...?