pauldix
Joined 16 years ago
Joined 16 years ago
It's a pretty neat algorithm from a paper in 2019 for the application "to index k-mers of DNA samples or q-grams from text documents". You can take a collection of bloom filters built for documents and then combine them together to have a single filter that will tell you which docs it maps to. Like an inverted index meets a bloom filter.
I'm using it in a totally different domain for an upcoming release in InfluxDB (time series database).
There's also code online here: https://github.com/bingmann/cobs
I'm excited to start doing some experimentation with Vortex to see how it can improve our products.
Great stuff, congrats to Will and team!
The 2.x to 3.x move is, admittedly, much harder. This is because of the language Flux. We haven't been able to bring that over to 3.x in a way that makes it useful. We actually built a bridge for it in our cloud offering, but our experience is that the performance isn't good enough to be acceptable for customers wanting to upgrade. If they want to make the move, adopting SQL or InfluxQL is likely the only path.
We'll continue to develop 3.x and we'll build more migration tooling over time. I think we can build specialized tooling to help Flux users migrate over to 3.x with query translation tools, but there are more features we need to land in 3.x to enable that first.
We're committed to the technology stack (Apache Arrow & DataFusion) and the 3.x line. We have no plans for another major release. I'll be happy if we end up releasing 3.56.2 8 years from now.
Then start over again to clean things out. It's not flawless, but it is surprising what it'll remember from a while back in the conversation.
I've been meaning to pick up some of the more automated tooling and editors, but for the phase of the project I'm in right now, it's unnecessary and the web UI or the Claude app are good enough for what I'm doing.
Right now I'm swapping between Gemini and Opus depending on the task. Gemini's 1M token context window is really unbeatable.
But the quality of what Opus 4 produces is really good.
edit: forgot to mention that this is all for Rust based work on InfluxDB 3, a fairly large and complex codebase. YMMV
We wrote and contributed the Rust object store crate used in this stack and by many others to the ASF.
This release is based on a "diskless" architecture that uses object storage for all durability. With DataFusion it has a columnar, vectorized, standards compliant SQL query engine. We also built support for InfluxQL on top of it.
The other big thing we brought in is an embedded Python VM using PyO3 and Python Build Standalone. This makes it possible to do data collection, ETL, monitoring, alerting, and all kinds of tasks inside the database at the point of collection.
Happy to answer any questions about the big project, what's next or anything time series related.
The InfluxDB part of it is more narrowly scoped than previous versions, but the Telegraf and Kapacitor parts are much more feature rich than those previous products.
Can you say more about your use case?