caerbannogwhite

Joined 6 months ago

Karma
56
Submissions
17
Member since
May 6
Computer engineer dressed as a stats programmer.

https://kolistat.com/

View on Hacker News

Recent submissions

Comment · 4 months ago
In reply to HackerThemAll's https://news.ycombinator.com/item?id=48659482 comment (which I think is a bit pretentious and unfair, and HN does not show my direct reply to the comment):

I think that's a red herring: the query is sandboxed DuckDB-WASM SQL (never executed as script), it's never injected into the DOM as HTML, and the page enforces a strict CSP that blocks inline script regardless. NoScript probably flags it because it's SQL-shaped text in a cross-site query string, and it matches its injection heuristic.

Comment · 4 months ago
Well done for laying that down in that way! I'm just wondering how much it would be on my side to support that, since I can see that you support cases like "count(*)" and "group by" inside visualize. I can see you have a full bison grammar, I only have a custom parser at the moment, but at least your implementation is in C. But I'm happy to follow thomasp85.
Comment · 4 months ago
Thank you for clarifying that! I don't think they got that and they also trusted NoScript a bit too blindly
Comment · 4 months ago
Thank you, that's quite a lot of great quality feedback!

Agreed, it's not production-grade yet, and robust standard errors are the priority. HC and cluster-robust SEs are the biggest credibility gap, which is why I just added an (Eigen-based) linalg kernel for the next release: it's the groundwork so the regression fitter can ship HC0–HC3 and clustered SEs (and bootstrapped, via the existing bootstrap aggregate). Margins and GLMs (IRLS on the same kernel, so not really a backend rewrite) are the next layers.

About the scope, agreed, and already narrowing. I'll credit posit ggsql as the real grammar-of-graphics tool; mine stays a minimal built-in. The plan is to go deep on the statistical core and lean on the DuckDB ecosystem for the rest.

For the market fit I'd frame it a bit differently. Not trying to pry anyone off Stata/R for rigorous work, I know that those aren't going anywhere soon. The niche is stats where the data already lives: people in SQL/DuckDB who currently round-trip to R/Python for a regression. R can run in the browser now (webR), but not in the same engine as the data. The stats-duck runs inside duckdb-wasm, so there's no separate runtime and no marshalling. And DuckDB is already far faster than R for the ETL/wrangling around the stats.

The plan is to figure out from real usage. For now, I'll focus on the core.

Comment · 4 months ago
Appreciate it, good call! I'll rename the ggsql bits and the docs to describe it as ggsql-inspired. I'll also point to ggsql for the real thing. Thanks for being kind about it!
Comment · 4 months ago
Yes, that's exactly its main purpose! I initially started because I needed a dataset browser. I work with clinical trials, so we usually get raw data files in all possible formats, from CSV to EXCEL and, of course, SAS formats. But since I was already using DuckDB, I thought about extending it a bit further, so you can quickly get a glance at the data.
Comment · 4 months ago
Thank you! I'll add note about it
Comment · 4 months ago
First of all, nice to meet you! The honest answer is timing: the first VISUALIZE commit on my side was april 25th; ggsql-duckdb's first commit was April 23rd. So I genuinely didn't know it existed!

About the name: yours is the official Posit one, and you were there first, so I'll rename my branding; there should be one ggsql, and it's yours. Mine only exposes VISUALIZE as the keyword anyway.

The actual name of the extension is the-stats-duck, which runs inside duckdb-wasm (it powers an in-browser data tool) and emits a Vega-Lite spec inline for the host to render. Your implementation (which I think is Rust based and an in-process HTTP server that opens a browser), is a native pattern, but correct me if I'm wrong! mine is deliberately thin and wasm-safe, not a whole engine.

About the parity, you're right, and I'm not chasing it; for real grammar-of-graphics, ggsql should be the tool! but, if that's ok with you, I'd love to keep the syntax aligned!

Comment · 4 months ago
That's exactly the inspiration! I made a post here on HN about that a few weeks ago: https://news.ycombinator.com/item?id=48108815

My plan is to release a blog post about all VISUALIZE current features next week, explicitly mentioning Posit's alpha GGPLOT.

/edit: clarifications

Comment · 5 months ago
(author here) I originally started this project to solve one of my main daily issues: having to quickly check data sparse in a bunch of datasets in different file formats (.xpt, .sas7bdat, .csv, .tsv, Excel, etc) and several folders.

I wanted it to - be performant: some files can have 50k rows or more, so an HTML table could not be the solution - be easy to configure: web-app, no installation or configuration, so my colleagues could use it too - (later) have a way to quickly explore data via queries: what's better than DuckDB for this?

Then, a few weeks ago, I read Thomas Lin Pedersen / Posit's alpha release here on HN (https://news.ycombinator.com/item?id=47833558), and I thought it could be nice to have charts in Bedevere. So I asked Claude to add the GGSQL grammar to my DuckDB extension (the-stats-duck - separate repo: https://github.com/caerbannogwhite/the-stats-duck).

Here are some other things you can try:

VISUALIZE bill_length_mm AS x , bill_depth_mm AS y , species AS color FROM penguins_clean DRAW point SCALE x DOMAIN 30 60 SCALE y ZERO false SCALE color TO viridis ;

VISUALIZE bill_length_mm AS x , bill_depth_mm AS y , species AS color FROM penguins_clean DRAW point FACET BY species COLS SCALE x ZERO false SCALE y ZERO false ;

Feedback is very welcome!

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton