asksol

I'm not sure operators are a good example, they can be idempotent or not depending on the implementation. The operator is a function that takes two values and return a result: result = op(left, right).

But that safety net doesn't extend to startup founders does it? AFAIR In Norway, you need to have worked in a company as a regular employee for at least one or two years before you can claim unemployment benefits. Health insurance is always there, and so is basic social benefits (but the unemployment one is the only you can live on).

In Faust we've had the ability to take a very different approach towards quality control. We have a cloud integration testing setup, that actually runs a series of Faust apps in production. We do chaos testing: randomly terminating workers, randomly blocking network to Kafka, and much more. All the while monitoring the health of the apps, and the consistency of the results that they produce.

If you have a Faust app that depends on a particular feature we strongly suggest you submit it as an integration test for us to run.

Hopefully some day Celery will be able to take the same approach, but running cloud servers cost money that the project does not have.

Yes, they do serve different purposes but they also share similarities. You could easily write a task queue on top of Faust.

It's important to remember that users had difficulty understanding the concepts behind Celery as well, perhaps it's more approachable now that you're used to it.

Using an asynchronous iterator for processing events enables us to maintain state. It's no longer just a callback that handles a single event, you could do things like "read 10 events at a time", or "wait for events on two separate topics and join them".

Faust is a library that you can import into your Python program, and all it requires is Kafka. Most other stream processing systems require additional infrastructure. Kafka Streams has similar goals, but Faust additionally enables you to use Python libraries and perform async I/O operations while processing the stream.

We had support for it initially, but we ended up not using the schema registry server. We want to add support for this back in if it provides value.

There's an examples/django project in the distribution. I think they removed the gevent bridge from PyPI for some reason, but you can still use the eventlet one. Gevent is production quality and the concept of bridging them is sound, so hope someone will work on it.

We added two levels of abstractions for this purpose:

- A Stream iterates over a channel

- A Channel implements: `channel.send` and `channel.__aiter__`.

- A topic is a "named" channel backed by a Kafka topic

- Further the topic is backed by a Transport

- Transport is very Kafka specific

To implement support for AMQP the idea is you only need to implement a custom channel.

If you open an issue we can consider how to best implement it.

Faust can serve the data over HTTP/websockets and other transports, so you can query it directly!

I'm looking at Redis Streams and it seems to lack the partitioning component of Kafka. One of the properties that we rely on is the ability to manually assign partitions such that a worker assigned partition 0 of one topic, will also be assigned partition 0 of other topics that it consumes from.

Simplicity is of course a goal, but this may mean we have to sacrifice some features when Redis Streams is used as a backend.

I'm glad you like Celery, this project in many ways realize what I wanted it to be.

My name is Ask and I'm the co-creator of this project, along with Vineet.

Yes, we're definitely interested in supporting Redis Streams!

Faust is designed to support different brokers, but Kafka is the only implementation as of now.

My name is Ask and I'm co-creator on this project, along with Vineet Goel.

This answer sums up why we wanted to use Python for this project: it's the most popular language for data science and people can learn it quickly.

Performance is not really a problem either, with Python 3 and asyncio we can process tens of thousands of events/s. I have seen 50k events/s, and there are still many optimizations that can be made.

That said, Faust is not just for data science. We use it to write backend services that serve WebSockets and HTTP from the same Faust worker instances that process the stream.

You don't think Equifax have lawyers? Of course they do, and they know that the clause exists. Just having the clause in the first place is a conspiracy to abuse consumers.

Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production.

Disclaimer: I'm a contributor