pacofvf
Joined 15 years ago
Joined 15 years ago
If you’ve tried Soda, you’ve probably noticed that many of the proper checks (like change over time, anomaly detection, etc.) are hidden behind their cloud product. Great Expectations, while powerful, can feel overly complex and brittle for modern analytics workflows. I wanted something that falls between lightweight, expressive, and flexible enough to integrate into any analytics stack.
Weiser is config-based; you define checks in YAML, and it runs them as SQL against your data warehouse. There’s no SaaS platform, no telemetry, no signup: just a CLI tool and some opinionated YAML.
Some examples of built-in checks:
1. Row count drops compared to a historical window
2. Unexpected nulls or category values
3. Distribution shifts
4. Anomaly detection
5. Cardinality changes
The framework is fully open-source (MIT license), and its goal is to be both human- and machine-readable. I’ve been using LLMs to help generate and refine Weiser configs, which work surprisingly well, far better than trying to wrangle pandas or SQL directly via prompt. I already have an MCP server that works well but it's a pain in the ass to install it in Claude Desktop, I don't want you to waste time doing that. Once Anthropic fixes their dxt format, I will release a MCP tool for Claude Desktop.
Currently it only supports PostgreSQL and Cube as datasource, and for destination for the checks results it supports postgres and duckdb(S3), I will add snowflake and databricks for datasources in the next few days. It doesn’t do orchestration, you can run it via cron, Airflow, GitHub Actions, whatever you want.
If you’ve ever duct-taped together DBT tests, SQL scripts, or ad hoc dashboards to catch data quality issues, Weiser might be helpful. I would love any feedback or ideas. It’s early days, but I’m trying to keep it clean and useful for both analysts and engineers. I'm also working on a better GUI.
GitHub: https://github.com/weiser-ai/weiser Docs: https://weiser.ai/docs/tutorial/getting-started
I'm happy to answer questions or hear about what other folks are doing to address this problem.
It's easier to explain what Cube is if we first define what the Semantic Layer(SL) is. In a few words, the SL is the abstract representation of business objects, for example: sales, users, conversion rates, etc. Cube provides the language to define the SL, an API to access it, access control mechanisms and a caching layer. It's important to emphasize that Cube is a stand-alone SL, decoupled from any BI visualization tool. That's the "headless" part, and I would also add that is "feetless" since it supports multiple source DBs. Looker the other big name in the space has the incentive of selling you more usage of BigQuery and of locking you in with their UI, it just recently started to open up to the idea of APIs. The idea is that you have a central place where you define the SL and then you don't need to duplicate the definition on every downstream application, which may lead to errors or inconsistencies.
> Is it that it can perform a single query across multiple databases?
Cube allows you to join data from multiple databases at the caching layer, that's fundamentally differently than a federated query engine. But from the downstream application perspective it has the same outcome. By being done at the caching layer it has inherent advantages and limitations vs federated queries.
I really like these series of articles by David Jayatillake that go into deeper detail:
1. https://davidsj.substack.com/p/semantic-superiority-part-1 2. https://davidsj.substack.com/p/semantic-superiority-part-2 3. https://davidsj.substack.com/p/semantic-superiority-part-3 4. https://davidsj.substack.com/p/semantic-superiority-part-4 5. https://davidsj.substack.com/p/semantic-superiority-part-5
LOL no.
First of all if you are doing "for i in range(N)" then you are already doing it wrong, for ML and data analytics you should be using NumPy "np.arange()", Numpy arange doesnt even run in "python" it's implemented in C. So it may even be faster than swift '..<' . Let me know when you can use swift with spark.
SinTrafico is a profitable and growing Mexican Startup. We are the leaders in the mobility industry in Mexico and soon in Latin America and beyond.
We are looking for senior developers on different roles and we are open to do remote work and also willing to help to relocate to Mexico City.
Stack: AWS, Python, Vue.js/React.
You should definitely apply if you have experience dealing with Spatial and Geographic data.
Please send me your CV at [email protected]