ersatz_username
Joined 5 years ago
Joined 5 years ago
Unless you think Sagan would have been surprised by the presence of an electric field (mediated by virtual particles…) between electrons and protons in an atom it’s quite likely you’re choosing an obtuse understanding of the term.
Here are some things we think are really important though
1. Data quality testing ideally happens during CI not after merge.
2. Developers come first. Virtually every aspect of the tool can be customized, modified, and extended down to the basic data model without changing any upstream core code. Want to build your own custom application on top of your data lineage? Great! Have at it!
3. Users should be able to own not just their own data but their own metadata. We go to great lengths to maintain feature parity between the cloud and self-hosted application.
We actually already do data monitoring as well although we haven't built the specific alerting features of Monte Carlo. There are quite a few tools that do that really well so it's not our focus at the moment.
More generally we can embed the transformation logic of each stage of your data pipelines into the edge between nodes (like two columns). Like you said, in the case of SQL there are lots of ways to statically analyze that pipeline but it becomes much more complicated with something like pure python.
As an intermediate solution you can manually curate data contracts or assertions about application behavior into Grai but these inevitably fall out of sync with the code.
Airflow has a really great API for exposing task level lineage but we've held off integrating it because we weren't sure how to convert that into robust column or field level lineage as well. How are y'all handling testing / observability at the moment?
The new Qwen3.8 release is really great and I know many people may not have GPUs that allow them to easily test it out so we are making it free to use through at least through the rest of the month.