mariusae
Joined 17 years ago
Karma
640
Submissions
40
Member since
Feb 26
Recent submissions
Comment · 1 month ago
Comment · 2 years ago
> Page and Brin […] and are often at each other's throat.
Citation needed.
Comment · 5 years ago
If you want to work with Go, check out bigslice [1] (and bigmachine [2]), which is built on a similar architecture.
[1] https://github.com/grailbio/bigslice/ [2] https://github.com/grailbio/bigmachine
Comment · 5 years ago
Reflow [1] is a similar attempt at a slightly different domain: bioinformatics and ETL pipelines. Reflow exposes a data model and programming model that reclaims programmability in these systems, and, by leaning on these abstractions, gives the runtime much more leeway to do interesting things. It unties the hands of the implementer.
Comment · 7 years ago
See also bigslice (https://bigslice.io) for another take on this.
Comment · 8 years ago
> I want an Airflow that runs and scales in the cloud
I'd encourage you to look at Reflow [1] which takes a different approach: it's entirely self-managing: you run Reflow like you would a normal programming language interpreter ("reflow run myjob.rf") and Reflow creates ephemeral nodes that scale elastically and that tear themselves down, only for the purpose of running the program.
> has extensive observability (monitoring, tracing)
Reflow includes a good amount of observability tools out of the box; we're also working on integrating tracing facilities (e.g., reporting progress to Amazon x-ray).
> has a full API, and maybe some clear way to test workflows.
Reflow's approach to testing is exactly like any other programming language: you write modules that can either be used in a "main" program, or else be used in tests.
Comment · 8 years ago
Reflow [1] is also well-suited for ETL workloads. It takes a different tack: it presents a DSL with data-flow semantics and first-class integration with Docker. The result is that you don't write graphs, instead you just write programs that, due to their semantics, can be automatically parallelized and distributed widely, all intermediate evaluations are memoized, and programs are evaluated in a fully incremental fashion:
Comment · 9 years ago
That's false. With anonymous closures you can implement select. See concurrent ML, or https://github.com/twitter/util/blob/develop/util-core/src/m... (which also happens to be lock free)
Comment · 13 years ago
yes, they were recorded, and will be posted.
1) Staying on top of my kid's school obligations, homework, etc. School Learning Management Systems are a hot mess, and I use agents to consolidate and collate, to keep track of the homework that has been done, what needs to be done, and even pre-grade homework to give feedback (most of the work is done on digital platforms too). It's been immensely time saving for these kinds of things. 2) Multiple instances of watching things passively: hotel room prices, availability of tickets, finding the best place to sell things online. In general things that require a lot of browsing, collating, and watching. These are the kinds of things that specialized software products could do as well, but with an agent they are just a prompt away, and you get exactly what you want. (More or less.)