dxtrous

A co-creator of Pathway here - noticed some inbound from the HN link and was a bit shocked to see the discussion. Of course, Airflow is one of the most widely used technologies out there, even if sometimes considered part of the "legacy" orchestration stack.

For software that is anything other than a modifiable template, stars and forks typically serve users as bookmarks. In the distant past, forks were the prevailing bookmark mechanism. Users who joined Github long after the star functionality was added generally prefer stars over forks for bookmarking. Personally, when evaluating current production use of a project, I index most strongly on the count and type of issues raised.

As for the Pathway framework specifically, if you dive in, the star-gazer demographics is split roughly evenly between: (A) folks, like parent, who heard about the tech one way or another and thought it was cool enough to engage with; and (B) participants of bootcamps and hackathons based on this framework and run in partnerships with top-tier universities globally. The license also restricts forking, and is less welcoming to external contributions than Apache projects.

Well spotted. Thank you!

Missed it - seems HN moderation (or OP?) changed the headline to the paper title.

> Claims/Observations end with a QED-symbol

Author comment: as a fairly common convention, QED immediately after a particular statement means that the statement should be considered proven. Depending on the text, this may either be because the statement (Observation) is self-explanatory, or, the discussion in the text leading up to the statement is sufficient, or, whenever the final statement of a Theorem follows as a direct corollary of Lemmas previously proven in the text.

From the authors: great question. If you take an "easy" task for long-range dependencies where a Mamba-like architecture flies (and the transformer doesn't, or gets messy), the hatchling should also be made to fly. For more ambitious benchmarks, give it a try in a place you care about. The paper is really vanilla and focused on explaining what's happening inside the model, but should be good enough as a starting point for architecture tweaks and experiments.

One of the authors here. I am genuinely thrilled to know someone would like to know what "B" stands for, knowing/expecting that "DH" stands for "Dragon Hatchling"!

A clarification is thus due. As indicated in the github repo: "BDH" stands for "Baby Dragon: Hatchling". Technically, "Hatchling" would perhaps be the version number.

For readers who find this discussion point interesting, I recommend considering for context: 1. The hardware naming patterns in the Hopper/Gracehopper architecture. 2. The attitude to acronyms taken in the CS Theory vs. Applied CS/AI community.

You can, of course, use the almost equivalent scientific-sounding Latin-derived term ("neuromorphic"), buy popcorn, and come back with it for a discussion about memristors.

It's mostly still data in the unstructured realm. One case is "messaging" data (live indexing of communications, social, news updates, etc.). Another case is data which was not originally text but can be easily transcribed into text with an "adapter" - this includes live audio/video/screenshot streams. For now Pathway works with discrete event streams, so audio transcription needs to be done upstream - e.g. by pairing up a live captioning service with Pathway. On the use case side, it tends to be less about interactive question/answer and more alerting handled with pre-registered queries ("alert me when X happens").

It's absolutely great to see you've figured out the details of it! Indeed, the repository comes with a mix of Python and Rust which need to be built together. We trust our users not to re-build the package with altered parameters for production use (given the package build from sources is slightly non-trivial and takes an hour or two, one cannot really get this wrong by accident...). Then, for learning and non-production use, the BSL under which Pathway is distributed, allows you to do almost anything with the code.

For indexing operators, there is some flexibility regarding the amount of internal operator state that is persisted. Say, in a stream-stream join structure, it's actually often faster to rebuild its state from its "boundary conditions" than persist it fully. For vector indexes, it is necessary to persist rather more of the internal state due to determinism issues (the next time the index is rebuilt, it could come back different, and could give different approximate results, which is bad). Currently, the HNSW implementation which is the basis of VectorStoreServer is still not fully integrated into the main Differential Dataflow organization, and has its own way of persisting/caching data "on the side". All in all, this part of the codebase is relatively young, and there is a fair amount of room for improvement.

(Adrian from the Pathway team here.) Indeed, everything is RAM-based, and persistence/cache relies on file backends. The precise backend to use is a code configuration parameter. S3 or local filesystem are the currently supported options. For documentation, see the user guide under Deployment -> Persistence.

I was hoping to see an LSMT implementation in bash and left slightly disappointed.

Would it work on a printer?