data_ders
Joined 8 years ago
Karma
411
Submissions
160
Member since
Apr 8
ask me about my DAG bod
View on Hacker NewsRecent submissions
Comment · 1 month ago
I love ggsql, the team behind it, ggplot and the tidyverse in general!
I've been mulling extending dbt Charts to support ggplot, and will likely reach out to the team about this in the next few weeks.
The difference is that Vega-Lite, ggsql, ggplot2 are designed to create a single chart. Yes, you can use facet_wrap() to great a series of charts based on a particular variable, but arguably that's still one chart!
What's missing that is table stakes for a BI dashboard: KPIs, text boxes, dropdowns, data sources, governance, and how the charts are arranged on the page and in relation to one another.
In the next few months we hope to ship a JSON schema that represents a dashboard in the same way that a VegaLite does for a single chart.
Does that make sense?
Comment · 1 month ago
(Anders here, on the dbt Charts team)
What drew me to the dbt project back in 2020 and what ultimately led me get a job at dbt Labs was this notion that when compared to software engineering teams, analytics teams had been vastly unserved by their tools.
In 2016 (and largely to this day), BI teams work in point-and-click interfaces without Software Development Lifecycle mainstays like source control, testing, and deployment environments.
There are BI engineers who do work like software engineers, but the gap b/w those who work like SWEs and those who don't is wide.
dbt Charts (like dbt before it did for data engineers) aims to empower data analysts to work with stakeholders in a more efficient and sustainable manner than was previously possible. The means to this end are: a succinct YAML DSL spec for defining charts, a powerful CLI that lets you validate, compile, and render the code into a dashboard spec the YAML.
Like data analysts being able to do PRs for changes to a dashboard is still rather unheard of especially if those PRs are in repos with other analytics code. Many BI vendors do ship some form of diffing and environment promotion, but virtually always these features fall short of what SWEs use every day.
Comment · 1 month ago
hey -- Anders here, I've been helping the dbt Charts team. I loved this list you've enumerated here
> There’s governance, access controls, interactivity, and connectivity to semantic layers
I'd add to that list as well: KPIs, dropdowns, text boxes, theming, shared data sources.
If you're familiar with Vega-Lite [1], I like to explain dbt Charts as the Vega-Lite but for BI, in that it is a language that specifies all of the components of BI.
> are those other pieces going to be part of dbtTran Cloud?
Certainly there are some features that are more conducive to being offered as a managed service. However, we've built this enterprise BI tool backwards in that we've started less with commercialization in mind, but rather laying the groundwork in the language and OSS project, so that we need not reinvent the wheel over and over again.
I'm hand-waving here about a future that doens't yet exist. But in theory dbt Charts could be extended in the front end to allow for not just SQL or SL metric support, but any query language. Same for the backend, we dbt Charts can render to png and html today, but other formats are also possible!
Comment · 5 months ago
I think the advantage is simplicity. Why connect first to duckdb and attach the db when you can query it directly with ADBC which is guaranteed to be fast
Comment · 5 months ago
Yeah for me standardization is the big win. But not just output formatting but cli commands and a guarantee that they’re as past as possible given that all the connectors use ADBC
Comment · 6 months ago
Reminds me of the common observance of “machine elves” when taking DMT
Comment · 6 months ago
thanks for the context!
> Auto-generating these dbt models and get the manifest aligned between Dagster code location
I just added you on LinkedIn. if you accept my connection there I can DM you a private preview document that you might find very interesting related to dbt project metadata (that is way less painful than `manifest.json`)
Comment · 6 months ago
hiya, anders from dbt here. cool project -- I especially love the branching and budgeting options you've built in. both are things that I'd love for the dbt standard to include one day. was it dbt's lack of those feature that inspired you to start this project? It also seems you have an aversion to Jinja, which, believe me, I get!
FYI dbt-fusion [1] is going GA next week (though GA for Databricks will come later) Most of it is source-available and ELv2-licensed, but there's a number of crates that are Apache 2.0, namely: dbt-xdbc, dbt-adapter, dbt-auth, dbt-jinja, dbt-agate. We also have plans to OSS more as time goes on (stay tuned).
I just wanted to call out the OSS crates in case you'd rather focus on "making your beer taste better" than have to re-build foundations. I'd love to hear if any of those crates come in handy for you (even more so if they don't work for you).
Feel free to reach out on LinkedIn or dbt community Slack if you ever want to chat more!
Comment · 6 months ago
plus 1 for ADBC!
Comment · 6 months ago
ok, this is definitely up my alley. color me nerd-sniped and forgive the onslaught of questions.
my questions are less about the syntax, which i'm largely familiar with knowing both SQL and ggplot.
i'm more interested in the backend architecture. Looking at the Cargo.toml [1], I was surprised to not see a visualization dependency like D3 or Vega. Is this intentional?
I'm certainly going to take this for a spin and I think this could be incredible for agentic analytics. I'm mostly curious right now what "deployment" looks like both currently in a utopian future.
utopia is easier -- what if databases supported it directly?!? but even then I think I'd rather have databases spit out an intermediate representation (IR) that could be handed to a viz engine, similar to how vega works. or perhaps the SQL is the IR?!
another question that arises from the question of composability: how distinct would a ggplot IR be from a metrics layer spec? could i use ggsql to create an IR that I then use R's ggplot to render (or vise versa maybe?)
as for the deployment story today, I'll likely learn most by doing (with agents). My experiment will be to kick off an agent to do something like: extract this dataset to S3 using dlt [2], model it using dbt [3], then use ggsql to visualize.
p.s. @thomasp85, I was a big fan of tidygraph back in the day [4]. love how small our data world is.
[1]: https://github.com/posit-dev/ggsql/blob/main/Cargo.toml
[2]: https://github.com/dlt-hub/dlt
[3]: https://github.com/dbt-labs/dbt-fusion
[4]: https://stackoverflow.com/questions/46466351/how-to-hide-unc...
Comment · 8 months ago
right? like it's a graph and a relational model query and a pipeline and a language and an abstract syntax tree and declarative logical plan
Comment · 8 months ago
what do you think is the "most bad" thing about SQL?