scott_s
Joined 18 years ago
Karma
34,207
Submissions
8,704
Member since
Mar 2
Recent submissions
Comment · 1 month ago
It is indeed interesting, and it's from Nathan Marz, who was the creator of Storm. That was the first major open source streaming platform. It predated what I consider the default open source streaming platform, Flink.
Glancing through the docs, my main three reactions are: 1) It's sophisticated system which, as you say, effectively opens up the components of a database engine to be used as needed. 2) Folks will still want to eventually land their data in a "normal" data-at-rest storage format like Hive tables or Parquet. 3) Folks will still want SQL.
Comment · 1 month ago
That work is still relevant! It's just that end-users don't need to be aware of it. The position of the paper I submitted, which I basically agree with, is that "streaming" shouldn't need to be something end-users care about. It's something the system does based on needs.
Databases already have a dataflow style architecture: that's how they implement queries. Because SQL is relational, SQL queries become dataflow execution plans.
One way to think about the programming model I worked on is that it was like exposing a query plan API directly to users, instead of giving them SQL.
Comment · 1 month ago
Agreed agreed.
During my time in the space, my pithy saying about the system I worked on [1] was that we could scale up or down. If you wanted to do streaming packet filtering with microsecond latency, we could do that. If you wanted to do complex analytics on structured data, we could do that. We did have deployments that "scaled down" and were more stream processing rather than streaming analytics. But analytics is by far the dominant use case, and SQL and relational databases are the better abstraction there. And for the stream processing cases, folks tend to stick to their existing lower-level stacks.
[1] I worked on IBM Streams, https://www.ibm.com/docs/en/streams/4.3.0?topic=welcome-intr..., which had its own language, compiler and runtime system. IBM sold this technology in 2023: https://21cs.com/en/resources/articles/2023/10/10/21cs-acqui....
Comment · 1 month ago
I worked in the streaming area for a decade, doing research and development (see: https://scholar.google.com/citations?user=Rdf5OIYAAAAJ&hl=en). After moving on from streaming specifically and moving into the general problems in large data warehouses, I also concluded: just default to SQL for all analytics and the database lens is the best way to think about streaming for analytics.
I still do think that stream programming models are extremely interesting and powerful. But I used to think they would eventually become more mainstream as a way to elegantly program for high throughput, low latency massively parallel systems. That has not been the case, and I no longer think that it will be. People get by with the existing programming languages and models, that seems to be fine.
Comment · 2 months ago
I have not been following, but some searching lead me to the conclusion that contracts are currently in the draft, but Stroustrup and some others are loudly saying that is a mistake. See: https://wrocpp.github.io/posts/contracts-dispute/
Comment · 2 months ago
A good place to start: https://news.ycombinator.com/item?id=46195226. (The submitter should be familiar.)
Comment · 2 months ago
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
Comment · 4 months ago
We dynamically load the shared object files. See: https://github.com/meta-pytorch/torchcodec/blob/8bbce656797c...
Solutions that shell out to the `ffmpeg` binary are not going to perform well.
Comment · 4 months ago
I was copying URLs fast-and-furious. The PyTorch Stable ABI reference is: https://docs.pytorch.org/docs/2.12/notes/libtorch_stable_abi...
Comment · 4 months ago
1. A higher-level API that better integrates into the PyTorch ecosystem.
2. Ease of going back-and-forth between CPU and GPU; in our experience, there's still a lot of scenarios where CPU decoding makes sense.
3. Audio decoding support.
Please take a look at our tutorials to get a feel for what TorchCodec can do: https://meta-pytorch.org/torchcodec/stable/generated_example...
Comment · 4 months ago
You can get older version of TorchCodec that work with older version of PyTorch, but it unfortunately will not have the new features (HDR video decoding; fast Wav decoding) in the latest release. See the compatbility matrix: https://github.com/meta-pytorch/torchcodec#compatibility-wit...
Up until recently, TorchCodec releases worked with one-and-only-one version of PyTorch. This is because up until recently, PyTorch did not have a stable ABI, and we needed to pin TorchCodec releases to PyTorch releases. But! PyTorch now has an excellent Stable ABI (https://github.com/meta-pytorch/torchcodec#compatibility-wit..., https://www.youtube.com/watch?v=HNdEmnvMvGE&t=1s) and TorchCodec is taking advantage of that since version 0.12.
Comment · 4 months ago
The one you have installed. :) We don't distribute FFmpeg and instead find your installed version at runtime. We support versions 4 through 8.
http://www.scott-a-s.com