followtherhythm

Hi, of course there are :) So the transformations are deployed in a separate NodeJs process per node. You can use familiar tools to take a look at its cpu + memory usage.

As for how it affects redpanda, currently we are working on exposing metrics via Prometheus. All of this work occurs on background threads, so the only way it would affect throughput is if enough coprocessors are deployed with enough actual work that would eventually start to fill up our seastar reactors.

Currently they cannot, as they share all of the same properties as their respective source topics, but this idea is good! I'm adding it to our list of design enhancements.

Good question! You can't re-key your records but this done is by design. We want the materialized partitions to reside on the same brokers as their sources so there is no network overhead for performing the transform, and it occurs on all nodes. So by choosing this design we traded off more CPU for lower latency and less network bandwidth.

We see a lot of demand for onboard transformations, and JS is our first trial, since most devs are familiar with it. We plan to look into Rust and all of the Wasm supported languages, based on user demand

Yup, as of now we are leveraging Nodejs for the technical preview. We also have another preview using embedded v8 for inline transforms, that is your transform runs in-process and apply() boils down to a function call. We are still evaluating other runtimes like wasmer as well, v8 was initially chosen for its maturity and community support.

Hi I'm one of the authors of the blog post, happy to answer any questions here !

This is so dope, I am trying it out asap!

Yes Storm is still under active development. Yahoo, Baidu and other big players continue to use Storm.

To be honest Storm's API is richer. Our approach however was to make stream processing available to all developers. It doesn't get simpler than four callbacks:

  void init(CtxPtr context);
  void destroy();
  void processRecord(CtxPtr context, FrameworkRecord &&r);
  void processTimer(CtxPtr context, const string &key, int time);
  Metadata metadata();

Since Mesos uses Zookeeper and we are tightly bound to Mesos, at the moment this is not possible.

There are a lot of reasons why, probably enough to write a blog post about. To summarize I would say that this is mainly a byproduct of being close to the metal, and our use of lock free code on the fast path. Also our system is relatively simple when compared to Storm which I also believe lends to less work and improved performance.

Hello, I am a software engineer at Concord. Happy to answer anyones questions