DataFusion is primarily a batch OLAP system, so we should be able to support hybrid workloads as well. And definitely agree with you re: Polars dev exp. That is something we are aiming for with our forthcoming Python sdk.
> It'd also be important/useful to support Python udfs (think numpy/jax/etc.).
Yep that's our longterm gameplan.
> It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.
Are there examples of project that do this? I'd be very much interested in looking into this.
> It'd also be important/useful to support Python udfs (think numpy/jax/etc.).
Yep that's our longterm gameplan.
> It'd be very cool if you could collaborate with or even tap into the polars frontend. If you could execute polars logical plans but with a streaming source, that would be huge.
Are there examples of project that do this? I'd be very much interested in looking into this.