secondrow
Joined 10 years ago
Joined 10 years ago
In this example, Xorq is used to compose an open source FeatureHouse that runs on DuckLake and interfaces via Apache Arrow Flight.
The post explains how: - The FeatureHouse is composed with Xorq - Feature leakage is avoided - The FeatureHouse can be ported to any underlying query engine (e.g., Iceberg) - Observability and lineage are handled - Feast can be integrated with it
But really, any system that runs the risk of failing or committing an error should have something in place to observe it, undo it, resume it. Your point about "big enough scale" is true - you can write your own code to handle that, and manually troubleshoot and repair corrupted data up to a certain point. But that takes time.
By making durable execution more lightweight/seamless (a la DBOS or Restate), the use of durable execution libs become just good programming practice for any application where cost of failure is a concern.